Home / Notes / VXLAN Tunnel Up→Down Runbook
NOTES · VXLAN EMERGENCY RESPONSE

VXLAN Tunnel Up→Down: An Emergency Runbook

hwNvo3VxlanTnlDown fires and a tunnel that was fine a minute ago is gone. This is the order that gets it back safely — what to confirm first, what to gather before you change anything, the two root causes that account for most of these alarms, and how to prove the fix actually worked before you close the ticket.

By the AtlasCommTech engineering team — 13 years of carrier & enterprise network deployments · Updated July 2026

Why Order Matters More Than Speed Here

The instinct when an alarm fires is to fix the first thing that looks wrong. On a VXLAN tunnel alarm, that instinct causes more outages than it prevents.

hwNvo3VxlanTnlDown means exactly one thing: a VXLAN tunnel that was up has gone down. It doesn't tell you why, and reacting to the alarm text alone — restarting a process, bouncing an interface, pushing a configuration change — is how a single-tunnel alarm turns into a wider outage. This runbook follows a fixed order: confirm what actually happened and how far it reaches, gather the evidence that tells root cause apart from noise, work the two causes that account for most of these tickets, and verify the specific tunnel recovered before calling it done.

What follows is that order with the exact commands, the one command that most often ends the incident — a targeted interface shutdown — and why it has to be the right interface, not just any interface that looks unstable.

The Runbook Has One Branch Point

Four steps, one branch point — worth having the picture in front of you before you touch anything.

The branch point is simple: is the route to the peer VTEP missing entirely, or is it there but flapping? The two answers lead to two completely different fixes, and running the wrong one wastes the minutes that matter most in an active outage.

Alarm: hwNvo3VxlanTnlDown Stage 1 · confirm + scope impact Stage 2 · gather evidence Stage 3a · route to peer VTEP is gonerouting / EVPN fault — not a shutdown fix Stage 3b · route is flappingshutdown the identified unstable interface Stage 4 · verify this tunnel, specifically

Diagram labels are kept in English for engineering clarity.

Everything above the branch point is confirmation and evidence-gathering, not remediation — resist fixing anything until you know which side of the branch you're on.

Working the Runbook in Order

Four stages: confirm and scope, gather evidence, apply the matching fix, verify the specific tunnel — not the network in general.

Stage 1 — Confirm the Alarm and Scope the Impact

Before touching any configuration, know what's actually broken and how far it reaches.

  1. Log in to the device named in the alarm — hwNvo3VxlanTnlDown carries the faulted device's IP or name directly, so use it rather than guessing which device is affected.
  2. Rule out a hardware fault first: check for a main control board or interface board fault on that device, and check the network management platform for the device showing as disconnected or carrying other active alarms. A board-level fault gets its own remediation and isn't a VXLAN problem at all.
  3. Confirm the impact is limited to VXLAN-carried traffic on this tunnel — that's the stated blast radius for this alarm, unless the hardware check in the previous step says otherwise.
%%01NVO3/4/hwNvo3VxlanTnlDown_active: The VXLAN tunnel changed from Up to Down.
(SourceAddress=4.4.4.100, DestinationAddress=3.3.3.100, TunnelId=4026531844)
// the alarm names the affected device and the tunnel's Source/Destination -> log in to that device, not a neighbor

<HUAWEI> display device
// rule out main control board / interface board faults before assuming this is a routing problem

Stage 2 — Gather the Evidence Before You Touch Anything

Two commands tell you which of the two root causes you're dealing with — run both before deciding on a fix.

  1. Run display ip routing-table toward the peer VTEP's address on the faulted device, to check whether the underlay route to the far-end VTEP is present at all.
  2. If the route is missing entirely, this is a route-loss case — go to Stage 3a.
  3. If the route is present, run display ospf peer brief (or the equivalent for your IGP) repeatedly, several seconds apart, and watch it across multiple runs — a neighbor that keeps appearing and disappearing between runs is flapping, and that's Stage 3b.
<HUAWEI> display ip routing-table 3.3.3.100
<HUAWEI>
// no route to the peer VTEP -> the underlay route is genuinely gone, go to Stage 3a

<HUAWEI> display ospf peer brief
 Peer Address     Interface           State
 10.0.12.2        GE1/0/1             Full
// run this again a few seconds later, and again --
// a neighbor that comes and goes between runs is flapping, not just slow to reconverge

Stage 3a — Root Cause: the Route to the Peer VTEP Is Gone

This is a genuine loss of path to the peer, not a flapping symptom — treat it as a routing/EVPN fault, not a tunnel fault.

  1. Work through the underlay route troubleshooting for the missing route directly — IGP/BGP configuration and neighbor state, not the VXLAN/NVE configuration.
  2. If the route was learned via EVPN and then withdrawn rather than lost at the underlay level, our VXLAN tunnel troubleshooting note's EVPN route-missing section covers exactly this failure mode in depth.

Stage 3b — Root Cause: a Route Is Flapping and Repeatedly Tearing the Tunnel Down

This is the fix that actually resolves most of these alarms — a small, deliberate, temporary cut, not a random interface bounce.

  1. From the repeated display ospf peer brief output, identify exactly which neighbor or interface is flapping — not just any interface that looks busy.
  2. Shut down that specific interface to break the unstable adjacency and stop it from repeatedly tearing down and rebuilding the tunnel.
  3. Treat this as a stopgap that stabilizes the tunnel, not a root-cause fix — the physical link, optic or upstream device causing the flap still needs its own investigation once the incident is stable.
[~HUAWEI] interface 10ge 1/0/1   // example only -- use the interface you identified as flapping
[~HUAWEI-10GE1/0/1] shutdown
[*HUAWEI-10GE1/0/1] commit

Stage 4 — Verify the Specific Tunnel, Not Just General Reachability

A working ping elsewhere on the fabric doesn't confirm this tunnel recovered — check the tunnel itself, by its Source/Destination pair.

  1. Run display vxlan tunnel and confirm State shows up for this specific tunnel, matched on the Source/Destination pair from the original alarm.
<HUAWEI> display vxlan tunnel
Tunnel ID   Source       Destination   State  Type      Uptime
4026531844  4.4.4.100    3.3.3.100     up     dynamic   00:00:42
// confirm State is up for this exact Source/Destination pair before closing the incident

5 Things That Go Wrong Even When You Follow the Runbook

Once the four stages above have told you where the incident sits, these five explain most of what still catches people out.

1. The Alarm Names the Symptom, Not the Cause — the Fix Is Almost Never on the Tunnel Itself

SYMPTOMhwNvo3VxlanTnlDown looks like a VXLAN problem, so the instinct is to go looking at VXLAN or NVE configuration first.

CAUSEThe tunnel is a passenger, not the cause — it goes down because the underlay stopped being able to reach the peer VTEP. Every real fix in this runbook happens in routing (a missing route or a flapping neighbor), never in the VXLAN/NVE configuration itself.

FIXStart evidence-gathering at the routing table, not the VXLAN configuration — display ip routing-table toward the peer VTEP is the first command that actually matters.

2. Shutting Down the Wrong Interface Turns One Outage Into Two

SYMPTOMThe tunnel comes back briefly after an interface shutdown, then the incident gets worse — more traffic affected, not less.

CAUSEInterface shutdown is a targeted, deliberate action against one specific flapping adjacency identified from repeated display ospf peer brief output — not a general “shut down anything that looks unstable” move. Shutting down the wrong interface can remove a working path instead of the flapping one.

FIXConfirm the flapping neighbor across multiple repeated command runs before shutting anything down, and shut down only that interface.

3. A Board-Level Hardware Fault Gets Chased as a Routing Problem

SYMPTOMThe route to the peer VTEP looks fine, OSPF isn't flapping, and yet the tunnel stays down or keeps re-flapping.

CAUSEA failing main control board or interface board can produce exactly this signature — intermittent forwarding failures that look like a routing problem from the CLI, when the actual fault is hardware. This is why Stage 1 checks board status and network-management alarms before Stage 2 even starts.

FIXIf Stage 2's evidence doesn't cleanly point to a missing or flapping route, go back and rule out a board fault via network management before spending more time on routing.

4. “The Tunnel Is Up” and “The Tunnel Is Fixed” Are Different Claims

SYMPTOMdisplay vxlan tunnel shows State up minutes after the fix, then the tunnel drops again.

CAUSEA tunnel recovering State up only confirms the immediate symptom cleared — it doesn't confirm the flapping neighbor identified in Stage 3b was actually the right one, or that its underlying cause (a bad optic, a failing link, an unstable upstream device) has been addressed. An interface shutdown is explicitly a stopgap in this runbook, not a close-out fix.

FIXTreat Stage 4's verification as confirming the incident is stable, not resolved — schedule the follow-up investigation into why that interface was flapping before the ticket is fully closed.

5. Not Every VXLAN Tunnel Fault Fires This Alarm

SYMPTOMA tunnel never came up in the first place — display vxlan tunnel shows Down from the start, with no Up-to-Down transition to alarm on.

CAUSEhwNvo3VxlanTnlDown specifically means a previously-up tunnel went down — it's an emergency response for a live, working tunnel that broke. A tunnel that never established at all is a different kind of fault, driven by the initial setup rather than something breaking a working state.

FIXFor a tunnel that never came up, use the dedicated VXLAN tunnel troubleshooting note's tunnel-establishment path instead of this emergency runbook — the causes and the fixes are different.

Related solution designs

Five Questions That Come Up Constantly

Pulled straight from the field — the ones worth having an answer ready for.

Isn't shutting down an interface during an outage exactly the kind of change you shouldn't make blind?

Yes — which is why Stage 3b only reaches for it after Stage 2 has already confirmed, across multiple repeated checks, exactly which neighbor is flapping. It's a targeted, evidence-based action on one identified interface, not a blind troubleshooting step, and it's explicitly the second thing this runbook tries, not the first.

What if display ip routing-table shows the route to the peer VTEP is present, but the tunnel still won't come back up?

Route presence confirms the underlay path exists, not that VXLAN itself is healthy — at that point this stops being a routing incident and becomes a VXLAN/EVPN establishment question. Our VXLAN tunnel troubleshooting note picks up from exactly this point.

Does this runbook replace the full VXLAN troubleshooting flow, or sit alongside it?

Alongside it, for a different moment. This runbook is for the specific emergency of an alarm firing on a tunnel that was working a moment ago — fast triage, evidence, a stopgap fix, verification. The full troubleshooting note is for the deeper root-cause work — EVPN route problems, VPN-Target mismatches, tunnel-establishment failures — that this runbook's evidence-gathering steps point you toward.

How do I tell a flapping neighbor from one that's just slow to reconverge after a real event?

Run display ospf peer brief several times in a row, a few seconds apart. A neighbor recovering from a real event settles into a stable state and stays there; a flapping neighbor keeps appearing and disappearing across your repeated runs. If you only run the command once, you can't tell the difference — that's exactly why Stage 2 calls for repeated checks, not a single look.

The tunnel is back up — what happens after I close the incident?

The interface you shut down in Stage 3b is still down, and it was carrying whatever the flapping condition actually was — a failing optic, a marginal link, an unstable upstream neighbor. That's follow-up work, not incident work: schedule the physical/optical investigation and don't bring the interface back up until you know why it was flapping in the first place.

Honest Limits of This Note

Honest Limits of This Note

This runbook is built around the hwNvo3VxlanTnlDown alarm and the two root causes — route loss and route flapping — that account for most of these emergency tickets. It assumes the tunnel was previously up and stable; a tunnel that never established at all is a different fault, covered in our VXLAN tunnel troubleshooting note, not here. It also doesn't replace a board-level hardware investigation, which Stage 1 tells you to rule out first but doesn't walk through in detail, and it doesn't cover controller-orchestrated recovery on SDN fabrics, where the same underlay logic applies but the commands differ.

In the middle of a VXLAN tunnel alarm right now?

Tell us what display ip routing-table and display ospf peer brief show for the affected tunnel, and we'll help you find the branch point fast.

WhatsApp an engineer →

Related Reading