Home / Notes / Router CPU & Memory Troubleshooting

NOTES · ROUTER CPU / MEMORY TROUBLESHOOTING

Router CPU & Memory High: From display health to Flame Graphs

An AR router pegged at high CPU, or slowly leaking memory, doesn't break the way a switch fighting a TC storm or an MSTP loop does — it's usually one process, one thread, or one growing memory handle. This is the command ladder that finds it: display health for the baseline, display cpu-usage top for the offending process, and — new since V600R024C10 — an actual flame graph for the case a process name alone doesn't explain.

By the AtlasCommTech engineering team — 13 years of carrier & enterprise network deployments · Updated July 2026

Why This Isn't the Switch Playbook

A router's control plane pegs for different reasons than a switch's — chasing the wrong culprit wastes the whole afternoon.

If the box in front of you is a switch and the CPU is pegged, that's very often a TC storm or an MSTP loop recomputing topology — see Switch CPU Utilization High: Finding What's Eating the Control Plane for that path. An AR router's CPU and memory complaints usually have a different shape: one process quietly running hot, one thread stuck in a loop, or one internal memory handle inside a process that keeps growing and never gives it back. None of that shows up as a topology storm, so the switch fault tree doesn't apply here.

What follows is the router-specific path: the baseline check, the process- and thread-level CPU drill-down with the exact commands, the new V600R024C10 flame-graph capture for the case the process name alone doesn't tell you enough, and the memory-side statistics and handle-alloc commands for a suspected leak — plus the gotchas and FAQ answers that come with actually running this in the field.

CPU or Memory — Two Different Drill-Downs From the Same Starting Point

Both start at display health. After that, the commands diverge completely.

display health is the single command that tells you which of the two problems you actually have, and on which slot — everything below branches from that one reading.

Diagram labels are kept in English for engineering clarity.

Notice the CPU side has a third stage the memory side doesn't: a live capture tool. That's the flame graph, and it only exists from V600R024C10 onward — on anything older, the thread callstack fields already inside display cpu-usage top are the fallback.

Working Through Each Stage

Four stages, the exact commands for each, and the real alarm text you'll actually see.

Stage 0 — Baseline: One Board, One Screen

display health puts CPU usage, memory usage and data-plane CPU usage for every slot on one screen — always the first command, regardless of which side of the problem you suspect.

<HUAWEI> display health
--------------------------------------------------------------------------------
Slot                        CPU Usage  Memory Usage(Used/Total)     DataPlane CPU Usage
--------------------------------------------------------------------------------
0     MPU(Master)           16%        32%   **82MB/**509MB         0%
--------------------------------------------------------------------------------

Stage 1 — CPU: Find the Process, Then the Thread

display cpu-usage top gives you the top processes and their threads in one pass, plus the thread's own call stack — worth running before assuming anything.

Run display cpu-usage top N from the diagnose view to see the top-N processes' CPU usage, and their busiest threads underneath each one.
Read the thread call stack the command prints alongside the numbers — it's what a support engineer would otherwise have to ask you for separately.
If you already have a process ID from a previous check, display cpu-usage slot cpu process processId drills one level further, into that one process's own threads.

<HUAWEI> system-view
[HUAWEI] diagnose
[HUAWEI-diagnose] display cpu-usage top 3
Info: This operation needs several seconds.
-------------------------------
Slot 0 CPU 0 information:
-------------------------------
Slot CPU use top 3 process:
-------------------------------------------------------------------
 ProcessID OsProcessID OsProcessname                          Usage
-------------------------------------------------------------------
  33176386       15170 fwmd                                      7%
  33177914       16698 pkimng                                    2%
      1000       18045 HSM                                       1%
-------------------------------------------------------------------
Slot CPU use top 3 thread:
---------------------------------------------------------------------------------------------------------
 ProcessID OsProcessID            Thread ID Thread Type                      Bind Cpu               Usage
---------------------------------------------------------------------------------------------------------
  33176422       15206                15361 PsspSchServer                    0-1                      14%
  33176386       15170                17351 fe_linkscan                      0-1                       3%
  33174969       13753                14231 vcmuagentPwr                     0-1                       2%
---------------------------------------------------------------------------------------------------------
Thread callstack information:
-------------------------------
Thread 0 [tid 15361] (Thread PsspSchServer):
#00  0x0000ffffa73e8f90 from [libc.so.6+0xe3f90](ioctl+0x10)
#01  0x0000ffff7d2d3c84 from [libpssp_sysdiag_daemon.so+0x46c84](GetCallStackInfoByTid+0x4c)
...

Stage 2 — Is This Live, or Already Cleared?

The default CPU alarm samples over an 8-minute window — by the time you're looking, the spike that triggered it may already be over. History and logs tell you which.

display alarm active (or display alarm history verbose) shows the raw hwCPUUtilizationRisingAlarm text, with the exact CpuUsage and CpuUsageThreshold values in the description field.
display cpu-usage peak history all, run from the diagnose view, shows the daily peak system and data-plane CPU usage for the last 15 days — useful for confirming whether this is a one-off or a recurring pattern.
Search the log buffer for SYSDIAG/6/PSSP_CPU_MODULE and DEBUG/4/DEBUG_CPUOVERLOAD entries — the latter prints the top-3 processes by name at the exact moment of overload, with each thread's own call stack underneath.

<HUAWEI> display alarm active
--------------------------------------------------------------------------------
Sequence   AlarmId    Severity Date Time  Description
--------------------------------------------------------------------------------
730        0xF10357   Critical 2025-03-26 The CPU usage exceeded the pre-set overload threshold. (TrapSeverity=3,Proba
                                02:48:52  bleCause=74299,EventType=3,PhysicalIndex=16777217,PhysicalName=MPU slot 0,Re
                                          lativeResource=CPU,UsageType=1,SubIndex=1,CpuUsage=95,Unit=1,CpuUsageThreshol
                                          d=90)
--------------------------------------------------------------------------------

<HUAWEI> system-view
[HUAWEI] diagnose
[HUAWEI-diagnose] display cpu-usage peak history all
Slot 0 CPU 0 peak cpu-usage last 15 days:
-------------------------------------------------------------------------------------------------
Index     System cpu-usage(%) SystemPeakDate          Dataplane cpu-usage(%) DataplanePeakDate
-------------------------------------------------------------------------------------------------
0                           5 2025-03-26 00:00:09                          0 2025-03-26 00:00:09
1                           5 2025-03-25 00:00:09                          0 2025-03-25 00:00:09
-------------------------------------------------------------------------------------------------

Stage 3 — When the Process Name Alone Doesn't Explain It: the Flame Graph

V600R024C10 added a live CPU-performance capture that outputs an actual flame-graph file — a genuinely new tool, not just another display command.

Start the capture with debug system cpu performance, specifying the slot, CPU, sampling frequency and duration — it warns explicitly that it may raise CPU and memory usage while it runs.
Collect the result with the matching collect sub-command once the capture window has elapsed.
The flame-graph file lands in the flash logfile directory as perf_${chassis-id}_${slot-id}_${cpu-id}.unfold — hand it to a support engineer to read; it isn't meant to be self-explanatory from the file name alone.
The same capture can also trigger itself automatically: once control-plane CPU overloads, or data-plane CPU reaches (overload threshold − 10)%, the device generates the flame graph on its own, no manual command needed.

<HUAWEI> system-view
[HUAWEI] diagnose
[~HUAWEI-diagnose]debug system cpu performance slot 0 cpu 0 frequency 100 time 20 core 1 process 1
Warning: This operation may cause high CPU and memory usage. Continue? [Y/N]:Y
Info: Operation succeeded.
[~HUAWEI-diagnose]debug system cpu performance slot 0 cpu 0 collect
Warning: This operation may cause high CPU and memory usage. Continue? [Y/N]:Y
Info: Operating, please wait for a moment............
Info: Operation succeeded.
// file: perf_${chassis-id}_${slot-id}_${cpu-id}.unfold, saved under flash logfile
// send this file to a support engineer for interpretation

Stage 4 — Memory: From the Process Table to the Handle That's Actually Growing

A high process-level memory number by itself isn't proof of a leak — the handle-level breakdown is what actually tells you.

display system memory statistics, run from the diagnose view, breaks total system memory down into process, filesystem and shared memory, then lists every process's own VmRSS and related fields.
Once a specific process looks suspicious, display system memory handle-alloc for that process ID lists every internal memory handle it holds and how much memory each one is using.
Watch the same handle across repeated checks — a handle whose UsedMemory only ever grows, and never shrinks back after the workload that raised it has passed, is the actual leak signature; collect the alarm, log and configuration information and escalate.

<HUAWEI> system-view
[HUAWEI] diagnose
[HUAWEI-diagnose] display system memory statistics slot 1 cpu 0
Slot 1 Cpu 0 memory utilization statistics at 2010-11-30 13:02:29 379 ms
------------------------------------------------------------------------------------------------------------------------------------
System Total                                                                                                 : 15592564 kB
System Used                                                                                                  : 5640984 kB
System Free                                                                                                  : 9951580 kB
------------------------------------------------------------------------------------------------------------------------------------
ProcessID      OsProcID       ProcName        VmSize    VmAs(Virtual Memory)      VmRSS      VmData    VmStk     VmExe     VmLib     Shmem     HugePage
-------------------------------------------------------------------------------------------------------------------------------------------------------
6              17275          CM              25000080  208124                    287192     207996    128       292       101504    2952720   1640448
33176386       15170          fwmd            164773728 299648                    265828     299520    128       744       117752    137730664 1640448

<HUAWEI> system-view
[HUAWEI] diagnose
[HUAWEI-diagnose] display system memory handle-alloc slot 0 cpu 0 process 883
----------------------------------------------------------------------------
HandleID HandleName                                          UsedMemory(B)
----------------------------------------------------------------------------
16 vos_queue                                               1,100,360
17 vos.simplemem.page.handle                               2,299,360
18 vos.simplemem.handle                                      216,016
19 v_vfs                                                     230,480
// re-check the same HandleID over time -- a value that only ever grows is the leak signature

6 Gotchas in the CPU/Memory Drill-Down Itself

The diagnostic commands have their own sharp edges — these are the ones that catch engineers off guard.

1. The Default Alarm Only Fires After an 8-Minute Sustained Overload

SYMPTOMA user complains the router "was slow for a minute," but no CPU alarm appears anywhere in display alarm active or the current alarm list.

CAUSEhwCPUUtilizationRisingAlarm samples CPU usage on a fixed cycle — 8 minutes by default — and only raises the alarm if every sample in that window stays above the overload threshold (90% by default). A spike that clears within the window never crosses that bar, even though it was real.

FIXDon't rely on the alarm list alone for a short-lived spike — cross-check display cpu-usage peak history all for the daily peak, and the log buffer for DEBUG/4/DEBUG_CPUOVERLOAD entries, which log the moment and the top-3 processes even when no formal alarm was raised.

StartTime   : 2025-03-26 02:48:52 ... CpuUsage=95 ... CpuUsageThreshold=90
ClearTime   : 2025-03-26 02:50:19 ... CpuUsage=5 ... CpuUsageThreshold=87
// alarm active for under 2 minutes here -- a shorter spike might not clear the 8-minute sampling window at all

2. Running the Diagnostic Commands Themselves Can Make a Loaded Box Worse

SYMPTOMCPU usage climbs further right after display cpu-usage top or display diagnostic-information is run on a box that's already struggling.

CAUSEBoth commands are explicitly documented as operations that can raise CPU usage while they run, and running them redundantly from several terminal sessions at once compounds the effect on an already-loaded device.

FIXReach for display cpu-usage top — the lighter, targeted command — before the much heavier display diagnostic-information bundle, and never run either one from more than one session on the same device at the same time.

3. The Flame Graph Is a New V600R024C10 Feature — and the File Still Needs a Human to Read It

SYMPTOMA perf_*.unfold file exists in the logfile directory, but nobody on-site can tell what it actually means.

CAUSEThe debug system cpu performance capture is only available from V600R024C10 onward, and the tool itself warns that running it can raise CPU and memory usage — it's meant for active investigation, not routine health checks. The resulting flame graph is a raw sampling artifact, not a labeled report; reading it correctly takes someone trained on the format.

FIXOn V600R024C10 and later, capture the flame graph only once you already suspect CPU is the problem, and send the .unfold file to a support engineer rather than trying to interpret it cold. On earlier versions, fall back to the thread callstack fields already printed by display cpu-usage top.

4. DEBUG_CPUOVERLOAD Only Prints the Top 3 Processes

SYMPTOMThe log shows three processes at the moment of overload, but the one you actually suspect isn't in the list at all.

CAUSEThe DEBUG/4/DEBUG_CPUOVERLOAD log entry is documented as printing only the top three processes by CPU occupancy at the moment of overload — a fourth-ranked offender simply doesn't make the cut.

FIXRun display cpu-usage top with a higher show-num directly, rather than relying on the log's fixed top-3 view, whenever the log's three names don't line up with what you expect.

5. A High Process Memory Number Isn't Proof of a Leak by Itself

SYMPTOMOne process's VmRSS in display system memory statistics looks large, and it's tempting to call it the leak straight away.

CAUSEA process can legitimately hold a large amount of memory for its normal working set. The actual leak signature only shows up one level down, in display system memory handle-alloc — a specific handle inside that process whose UsedMemory keeps climbing and never returns to baseline once the workload that raised it has passed.

FIXBefore escalating a "memory leak," re-check the same process's handle-alloc output across more than one sample and confirm at least one handle is monotonically growing — not just that the process total looks big once.

6. On a Dual-Main-Control Chassis, Each Board Keeps Its Own CPU/Memory State

SYMPTOMdisplay health and display cpu-usage on the active board look completely normal, but the reported symptom (a failover, a slowdown) doesn't match.

CAUSEEvery command in this note run on the active main-control board only reports that board's own CPU and memory — the standby board keeps entirely separate state and has to be reached on its own to confirm whether it was the one under load. This is the same caveat covered in more detail in Fault Information to Collect Before You Escalate.

FIXFrom the diagnose view, use local-telnet slave to reach the standby board and re-run display health / display cpu-usage top there as well, before concluding the active board's readings are the whole picture.

<HUAWEI> diagnose
[HUAWEI-diagnose] local-telnet slave
<HUAWEI> display health

Six Questions That Come Up Constantly

Pulled straight from the field — the ones worth having an answer ready for.

What's the default CPU alarm threshold, and how long before it actually fires?

The default overload threshold is 90% CPU usage, sampled on an 8-minute cycle by hwCPUUtilizationRisingAlarm — the alarm only raises if every sample in that window stays above 90%. The clear threshold sits a little below it (87% in a typical real case), so a spike can come and go faster than the alarm mechanism itself.

What's the actual difference between display cpu-usage and display cpu-usage top?

display cpu-usage (and display health) give you the board-level percentage — useful for confirming there's a problem at all. display cpu-usage top, run from the diagnose view, breaks that number down into the actual top processes and threads responsible, plus each thread's own call stack — it's the command that answers "which one."

I already have a process name from cpu-usage top — do I still need the flame graph?

Not always. Reach for the flame graph specifically when the process name alone doesn't explain the load — a process that's legitimately busy for a reason you understand doesn't need one. It earns its keep on the cases where the top process/thread view keeps pointing somewhere that doesn't make sense on its own.

Is a high number in display system memory statistics' process table itself the problem?

Not by itself — see the gotcha above. A large VmRSS can be a process's normal working set. The actual leak signature is one level down, in display system memory handle-alloc for that process: a specific handle whose UsedMemory keeps climbing across repeated checks and never comes back down.

My device is a switch, not a router — is this the same fault tree?

No. A switch's high-CPU tickets are much more often a TC storm or an MSTP loop recomputing topology than a single runaway process — see Switch CPU Utilization High: Finding What's Eating the Control Plane for that specific path instead of this one.

Which router software versions actually support the flame graph?

The debug system cpu performance capture and the resulting flame-graph file are new from V600R024C10 onward. On anything earlier, there's no flame graph to fall back on — the thread callstack fields printed directly inside display cpu-usage top are the equivalent tool for those versions.

Honest Limits of This Note

Honest Limits of This Note

This note is built around the Huawei AR-series router's V600 CPU/memory troubleshooting model and its display health / cpu-usage / memory statistics / handle-alloc commands, plus the V600R024C10 flame-graph capture, taken from Huawei's own AR maintenance handbook. It doesn't cover a specific list of memory-leak root causes — a growing handle always still needs a support engineer to read the flame graph or handle-alloc output and confirm what's actually holding the memory. On software older than V600R024C10, treat the flame-graph steps as not applicable and rely on the thread callstack view instead.

Reading a CPU or memory dump and stuck?

Send us your display health / cpu-usage top output, or the flame-graph .unfold file, and we'll help you read it.

WhatsApp an engineer →