Date: Sat, 18 Jan 2025 19:24:57 -0600 (CST) From: Timothy Pearson <tpearson@raptorengineering.com> To: freebsd-net <freebsd-net@FreeBSD.org> Subject: Re: FreeBSD 13: IPSec netisr overload causes unrelated packet loss Message-ID: <1820780643.6432954.1737249897808.JavaMail.zimbra@raptorengineeringinc.com> In-Reply-To: <2079636793.6405429.1737238589082.JavaMail.zimbra@raptorengineeringinc.com> References: <2079636793.6405429.1737238589082.JavaMail.zimbra@raptorengineeringinc.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Quick update --tried the IPSec deferred update patch [1], no change. A few tunables I forgot to include as well: net.route.netisr_maxqlen: 256 net.isr.numthreads: 32 net.isr.maxprot: 16 net.isr.defaultqlimit: 256 net.isr.maxqlimit: 10240 net.isr.bindthreads: 1 net.isr.maxthreads: 32 net.isr.dispatch: direct [1] https://www.mail-archive.com/freebsd-net@freebsd.org/msg64742.html ----- Original Message ----- > From: "Timothy Pearson" <tpearson@raptorengineeringinc.com> > To: "freebsd-net" <freebsd-net@FreeBSD.org> > Sent: Saturday, January 18, 2025 4:16:29 PM > Subject: FreeBSD 13: IPSec netisr overload causes unrelated packet loss > Hi all, > > I've been pulling my hair out over a rather interesting problem that I've traced > into an interaction between IPSec and the rest of the network stack. I'm not > sure if this is a bug or if there's a tunable I'm missing somewhere, so here > goes... > > We have a pf-based multi-CPU firewall running FreeBSD 13.x with multiple subnets > directly attached, one per NIC, as well as multiple IPSec tunnels to remote > sites alongside a UDP multicast proxy system (this becomes important later). > For the most part the setup works very well, however we have discovered > through extensive trial and error / debugging that we can induce major packet > loss on the firewall host itself by simply flooding the system with small IPSec > packets (high PPS, low bandwidth). > > The aforementioned (custom) multicast UDP proxy is an excellent canary for the > problem, as it checks for and reports any dropped packets in the receive data > stream. Normally, there are no dropped packets even with saturated links on > any of the local interfaces or when *sending* high packet rates over IPsec. As > soon as high packet rates are *received* over IPsec, the following happens: > > 1.) netisr on one core only goes to 100% interrupt load > 2.) net.inet.ip.intr_queue_drops starts incrementing rapidly > 3.) The multicast receiver, which only receives traffic from one of the *local* > interfaces (not any of the IPsec tunnels), begins to see packet loss despite > more than adequate buffers in place with no buffer overflows in the UDP stack / > application buffering. The packets are simply never received by the kernel UDP > stack. > 4.) Other applications (e.g. NTP) start to see sporadic packet loss as well, > again on local traffic not over IPsec. > > As soon as the IPSec receive traffic is lowered enough to get the netisr > interrupt load below 100% on the one CPU core, everything recovers and > functions normally. Note this has to be done by lowering the IPSec transmit > rate on the remote system, there is no way I have discovered to "protect" the > receiver from this kind of overload. > > While I would expect packet loss in an overloaded IPSec link scenario like this > just due to the decryption not keeping up, I would also expect that loss to be > confined to the IPSec tunnel. It should not spider out into the rest of the > system and start affecting all of the other applications and > routing/firewalling on the box -- this is what was miserable to debug, as the > IPSec link was originally only hitting the PPS limits described above > sporadically during overnight batch processing. Now that I know what's going > on, I can provoke easily with iperf3 in UDP mode. On the boxes we are using, > the limit seems to be around 50kPPS before we hit 100% netisr CPU load -- this > limit is *much* lower with async crypto turned off. > > Important tunables already set: > > net.inet.ipsec.async_crypto=1 (turning this off just makes the symptoms appear > at lower PPS rates) > net.isr.dispatch=direct (deferred or hybrid does nothing to change the symptoms) > net.inet.ip.intr_queue_maxlen=4096 > > Thoughts are welcome...if there's any way to stop the "spread" of the loss I'm > all ears. It seems that somehow the IPSec traffic (perhaps by nature of its > lengthy decryption process) is able to grab an unfair share of netisr queue 0, > and that interferes with the other traffic. If there was a way to move the > IPSec decryption to another netisr queue, that might fix the problem, but I > don't see any tunables to do so. > > Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1820780643.6432954.1737249897808.JavaMail.zimbra>