Gateway: OOM Killer Invoked for FRR zebra

[pau-hedgehog]

https://github.com/githedgehog/fabricator/actions/runs/20954875728/job/60217437206

CI passes but artifacts contain OOM kill logs:

grep oom 1-current/show-tech-output/basic-vpc-peering-suite/suite-setup/gateway-1-show-tech.log
root          62  0.0  0.0      0     0 ?        S    11:36   0:00 [oom_reaper]
Jan 13 12:12:26 gateway-1 kernel: bgpd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=1000
Jan 13 12:12:26 gateway-1 kernel:  oom_kill_process.cold+0x8/0x8d
Jan 13 12:12:26 gateway-1 kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Jan 13 12:12:26 gateway-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=cri-containerd-6c6baea31578ded9b2fbb696ffeecc142b8fd76c457b5f6a78c6f2275ac5dc3d.scope,mems_allowed=0,global_oom,task_memcg=/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd1bee17_c7dc_47e7_a65a_a98b2e9d6436.slice/cri-containerd-6c6baea31578ded9b2fbb696ffeecc142b8fd76c457b5f6a78c6f2275ac5dc3d.scope,task=zebra,pid=9709,uid=100
Jan 13 12:12:26 gateway-1 kernel: Out of memory: Killed process 9709 (zebra) total-vm:5903264kB, anon-rss:5433564kB, file-rss:64kB, shmem-rss:0kB, UID:100 pgtables:10768kB oom_score_adj:1000
Jan 13 12:12:26 gateway-1 kernel: Tasks in /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd1bee17_c7dc_47e7_a65a_a98b2e9d6436.slice/cri-containerd-6c6baea31578ded9b2fbb696ffeecc142b8fd76c457b5f6a78c6f2275ac5dc3d.scope are going to be killed due to memory.oom.group set
Jan 13 12:12:26 gateway-1 kernel: Out of memory: Killed process 9639 (tini) total-vm:2656kB, anon-rss:0kB, file-rss:240kB, shmem-rss:0kB, UID:0 pgtables:44kB oom_score_adj:1000
Jan 13 12:12:26 gateway-1 kernel: Out of memory: Killed process 9666 (docker-start) total-vm:4828kB, anon-rss:384kB, file-rss:108kB, shmem-rss:0kB, UID:0 pgtables:48kB oom_score_adj:1000
Jan 13 12:12:26 gateway-1 kernel: Out of memory: Killed process 9723 (bgpd) total-vm:256132kB, anon-rss:20956kB, file-rss:356kB, shmem-rss:0kB, UID:100 pgtables:124kB oom_score_adj:1000

gateway-1-show-tech.log

First observed by @qmonnet . Raising this to discuss with @Frostman if it's expected in CI or should be tracked and addressed

[Frostman]

@pau-hedgehog definitely not expecting OOMs in CI :)

[pau-hedgehog]

I tried increasing the GW VM RAM in HLAB but still seeing the OOM. The zebra process is just too much. I think it makes sense to move this issue to gateway repo
https://github.com/githedgehog/fabricator/actions/runs/21013028702/job/60412707872

Jan 15 00:00:22 gateway-1 kernel: Out of memory: Killed process 7617 (zebra) total-vm:5803120kB, anon-rss:5333460kB, file-rss:388kB, shmem-rss:0kB, UID:100 pgtables:10572kB oom_score_adj:1000

gateway-1-show-tech.log
control-1-show-tech.log

CC: @Fredi-raspall

[pau-hedgehog]

Yes, no matter the size of the GW memory, zebra grows as big as it can:
https://github.com/githedgehog/fabricator/actions/runs/21042697664/job/60509435363

Jan 16 00:13:44 gateway-1 kernel: Out of memory: Killed process 10145 (zebra) total-vm:16045504kB, anon-rss:15575892kB, file-rss:284kB, shmem-rss:0kB, UID:100 pgtables:30608kB oom_score_adj:1000

[Frostman]

@Fredi-raspall not sure I can do anything with it, no idea how zebra works. I'd suggest looking into some ways to specify to it how much memory it can consume. Let me know if you'll have some ideas, but I'll remove myself from it for now

[Fredi-raspall]

@Fredi-raspall not sure I can do anything with it, no idea how zebra works. I'd suggest looking into some ways to specify to it how much memory it can consume. Let me know if you'll have some ideas, but I'll remove myself from it for now

Sorry, I thought you had mentioned that it was not an FRR issue but something related to the environment. By looking at the logs provided by @pau-hedgehog , I realized that it wasn't and re-assigned it to me. Sorry for the noise.

[pau-hedgehog]

I took a look at: https://github.com/githedgehog/fabricator/actions/runs/21449034979/job/61773361800

I'm afraid the issue is still present. I noticed the show tech for gateway is huge!

pau@pau-TP-P14s:~/Downloads/fab-21449034979-h-gw-iso-l2vni/1-current/show-tech-output$ ls -lah gateway-1-show-tech.log 
-rw-r--r-- 1 pau pau 2.3G Jan 28 19:20 gateway-1-show-tech.log
pau@pau-TP-P14s:~/Downloads/fab-21449034979-h-gw-iso-l2vni/1-current/show-tech-output$ wc -l gateway-1-show-tech.log 
30985080 gateway-1-show-tech.log
pau@pau-TP-P14s:~/Downloads/fab-21449034979-h-gw-iso-l2vni/1-current/show-tech-output$ grep oom-kill gateway-1-show-tech.log 
Jan 28 19:15:12 gateway-1 kernel: bgpd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=1000
Jan 28 19:15:12 gateway-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=cri-containerd-de6a7f48755fc29988816208991fd2258bad72c8e44125b964f48d316565148a.scope,mems_allowed=0,global_oom,task_memcg=/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4cbb0d2_6b80_446f_9c4e_92188dd0a452.slice/cri-containerd-de6a7f48755fc29988816208991fd2258bad72c8e44125b964f48d316565148a.scope,task=zebra,pid=7070,uid=100

Most of the lines are like:

pau@pau-TP-P14s:~/Downloads/fab-21449034979-h-gw-iso-l2vni/1-current/show-tech-output$ grep native gateway-1-show-tech.log | wc -l
30980830

Could the culprit be running running vtysh -c 'show memory' ?:

--- qmem Native message allocations ---
Type                          : Current#   Size       Total     Max#  MaxBytes
native notify msg             :        0 variable         0        1       296
native subscribe msg          :        0 variable         0        1        88
native txn-reply msg          :        0     48           0        1        56
native cfg-reply msg          :        0     48           0        1        56
native cfg-act-reply msg      :        0     48           0        1        56
native notify msg             :        0 variable         0        1       296
native subscribe msg          :        0 variable         0        1        88
native txn-reply msg          :        0     48           0        1        56
native cfg-reply msg          :        0     48           0        1        56
native cfg-act-reply msg      :        0     48           0        1        56
native notify msg             :        0 variable         0        1       296
native subscribe msg          :        0 variable         0        1        88
native txn-reply msg          :        0     48           0        1        56
native cfg-reply msg          :        0     48           0        1        56
native cfg-act-reply msg      :        0     48           0        1        56
native notify msg             :        0 variable         0        1       296
native subscribe msg          :        0 variable         0        1        88
native txn-reply msg          :        0     48           0        1        56
native cfg-reply msg          :        0     48           0        1        56
native cfg-act-reply msg      :        0     48           0        1        56
native notify msg             :        0 variable         0        1       296
native subscribe msg          :        0 variable         0        1        88
native txn-reply msg          :        0     48           0        1        56
...

[Fredi-raspall]

2.3 GB of show-tech output for GW? This is definitely a red flag.