Date: Wed, 15 Jan 2020 10:14:31 -0500 From: mike tancsa <mike@sentex.net> To: John Jasen <jjasen@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: unexplained latency, interrupt spikes and loss of throughput on FreeBSD router/firewall system Message-ID: <4f7207fa-1ca8-df40-1c43-1c7ccfdf9afa@sentex.net> In-Reply-To: <CAACLuR0AYBSPajzmp9%2BaAK%2B02M6_pnai3b9s7jDbtXLvd1fGNw@mail.gmail.com> References: <CAACLuR0AYBSPajzmp9%2BaAK%2B02M6_pnai3b9s7jDbtXLvd1fGNw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/15/2020 9:55 AM, John Jasen wrote: > Executive summary: > > Periodically, load will spike on network interrupts on one of our > firewalls. Latency will quickly climb to the point that things are > unresponsive, sessions will timeout, and bandwidth will plummet. A couple of wild stabs... Are the routers generating any odd amount of ICMP response traffic at the time ? e.g. port|host unreachable etc ? (maybe track netstat -s -p icmp). Are there any bursts of icmp redirects happening ? I know that can slog a router sometimes-- Try instrumenting the appropriate oids (sysctl -a | grep -i redirect) to see if thats the case. A lot of small packets ? If possible maybe a network tap in front of the boxes to capture / profile the traffic before/after to see if there is something like a big scan happening or DOS with many small packets etc. If thats not possible, do you have enough spare CPU to do some netflow analysis on the box ? Or maybe take some periodic snapshots of the interface stats and compare normal to bad periods via sysctl -A dev.cxl | grep "_frames_" Good luck! ---Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4f7207fa-1ca8-df40-1c43-1c7ccfdf9afa>