Date: Wed, 07 Feb 2024 15:25:11 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 276870] mbuf cluster leak with on pf+bird2 bgp routers Message-ID: <bug-276870-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276870 Bug ID: 276870 Summary: mbuf cluster leak with on pf+bird2 bgp routers Product: Base System Version: 13.2-STABLE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: thomas@gibfest.dk Created attachment 248234 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D248234&action= =3Dedit Screenshot of mbuf cluster total use as reported by netstat -m over time Hello =F0=9F=99=82 Last month I had one of my FreeBSD routers stop forwarding (stopped respond= ing on the network at all, had to IPMI in) because it ran out of mbuf clusters.= It usually operates far from the limit, but there is (was) something leaking m= buf clusters bad, and I suspect it might be bird2, or a combination of bird2 an= d a FreeBSD kernel bug. ---- Some background: The boxes in question are BGP routers for a small network, they run bird2 a= nd only get a default route from upstream BGP, not a full table. Due to a missing/misconfigured kernel export filter bird was repeatedly try= ing to export some routes to the kernel which the kernel already knew (from statically configured blackhole routes). So these errors have been repeatin= g in the logs for some time (more than a year, meaning this in itself has not be= en an issue): Jan 11 19:09:04 dgncr2a bird[30963]: KRT: Error sending route 2a09:94c0::/2= 9 to kernel: File exists Jan 11 19:10:04 dgncr2a syslogd: last message repeated 1 times Jan 11 19:10:04 dgncr2a bird[30963]: KRT: Error sending route 85.209.116.0/= 22 to kernel: File exists Jan 11 19:11:04 dgncr2a syslogd: last message repeated 1 times Over the holidays I upgraded from bird 2.0.9 to bird 2.14, as well as upgra= ding FreeBSD from 13-STABLE-384a885111ad to 13-STABLE-2cbd132986a7. I suspect on= e of these two changes made this problem appear. I made no changes to bird or ro= uter config other than the upgrades. ---- The mbuf cluster leak was pretty bad, like 8-10 clusters per second at a pr= etty steady rate. The kern.ipc.nmbclusters limit on my routers was around 2 mill= ion and I raised it to 4 million now. Since I had no idea what was causing the leak and I was desperate for a fix= I at one point tried adding the missing kernel export filter (as to at least silence the noisy warnings in the logs), and imagine my surprise when the m= buf cluster leak stopped. I tried removing the filers again, the leak started again, and stopped again when I re-added the filters. It appears some combination of bird 2.14 and exporting routes already found in the kernel means leaking mbuf clusters li= ke crazy. I have no idea if this is a bird or a freebsd problem. I reported the issue= to the bird-users@ list http://trubka.network.cz/pipermail/bird-users/2024-January/017314.html and = was encouraged in that thread to open this PR as well. The attached grafana screenshot shows the per-second rate of increase (seen over 5 minutes) of the "total" number in the "mbuf clusters in use" line of= the `netstat -m` output for both routers. The green line is the active and the yellow line is the passive router. The drop in the green line and the following spike towards the end (2000-2100ish) is me filtering the blackhole routes from the bird kernel export, removing the filter to confirm, and re-adding it. I can to some extent test stuff, but the routers are in production so nothi= ng too wild. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-276870-227>