Date: Wed, 15 Jan 2020 14:24:14 -0800 From: Navdeep Parhar <nparhar@gmail.com> To: John Jasen <jjasen@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: unexplained latency, interrupt spikes and loss of throughput on FreeBSD router/firewall system Message-ID: <aadc3f1b-e4b9-7413-369f-e084619d36ce@gmail.com> In-Reply-To: <CAACLuR0AYBSPajzmp9%2BaAK%2B02M6_pnai3b9s7jDbtXLvd1fGNw@mail.gmail.com> References: <CAACLuR0AYBSPajzmp9%2BaAK%2B02M6_pnai3b9s7jDbtXLvd1fGNw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/15/20 6:55 AM, John Jasen wrote: > Executive summary: > > Periodically, load will spike on network interrupts on one of our > firewalls. Latency will quickly climb to the point that things are > unresponsive, sessions will timeout, and bandwidth will plummet. Is this with 9000 MTU? Can you please post "netstat -m" from this system? Assuming this is 9000 MTU, try setting this in /boot/loader.conf and reboot: hw.cxgbe.largest_rx_cluster=4096 > We do not see increases in ethernet pause frames, drops, errors, or > anything else like that from the system. This part is strange. The incoming frames are either being dropped (errors or overflows) or getting throttled via pause frames. I'd have expected "netstat -dI <ifnet>" to show errors or drops or "sysctl dev.cc dev.cxl | grep pause" to show some activity. Can you please double check? Regards, Navdeep > > Usually, the quickest fix is to failover to the backup firewall. At that > time, the backup firewall behaves normally and interrupt load drops on the > afflicted firewall device. > > I'm stumped. Networking says its these systems. I believe its something on > other side. > > Any ideas? > > Background information: > FreeBSD 11.3-RELEASE-p3 > hw.machine: amd64 > hw.model: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz > hw.ncpu: 24 > hw.machine_arch: amd64 > Firewall: pf > failover: CARP > network cards: seen with Chelsio T5-580 and T6 series cards. > other networking information: VLANs are in use. Occasional LAGG usage as > well. > > When this occurs, some of the interrupts dedicated to cxgbe queues spike > to 100%. Latency climbs to the point that TCP timeouts start kicking in, > and users start complaining. Bandwidth drops from 2-3Gbs to ~100-200Mbs > > netstat shows no increase of error or drop packets. sysctl shows no > increase in pause frames. > > I'm happy to provide further information. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?aadc3f1b-e4b9-7413-369f-e084619d36ce>