From owner-freebsd-net@freebsd.org Wed Jan 15 15:14:33 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 57F181F60BF for ; Wed, 15 Jan 2020 15:14:33 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47yW883tBvz4bsf for ; Wed, 15 Jan 2020 15:14:32 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:9144:94f1:31aa:b9bf] ([IPv6:2607:f3e0:0:4:9144:94f1:31aa:b9bf]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 00FFEVC0033499 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 15 Jan 2020 10:14:31 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: unexplained latency, interrupt spikes and loss of throughput on FreeBSD router/firewall system To: John Jasen , FreeBSD Net References: From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWibkB DQRcsMzkAQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4 axtKRSG1t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1 qzAJweEtRdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6c Lm0EiHPOl5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5 o9KKu4O7gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQAB iQE8BBgBCAAmFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAlywzOQCGwwFCQHhM4AACgkQeVOE Fl5WrMhmjQf/dBCjAVn1J0GzSsHiLvSAQz1cchbdy8LD0Tnpzjgp5KLU7sNojbI8vqt4yKAi cayI88j8+xxNXPMWM4pHELuUuVHS5XTpHa/wwulUtI5w/zyKlUDsIvqTPZLUEwH7DfNBueVM WyNaIjV2kxSmM8rNMC+RkgyfbjGLCkmWsMRVuLIUYpl5D9WHmenUbiErlKU2KvEEXEg/aLKq 3m/AdM9RAYsP9O4l+sAZEfyYoNJzDhTZMzn/9Q0uFPLK9smDQh4WBTFaApveVJPHRKmHPoNF Xxj+yScYdQ4SKH34WnhNSELvnZQ3ulH5tpASmm0w+GxfZqSc8+QCwoKtBRDUxoE56A== Message-ID: <4f7207fa-1ca8-df40-1c43-1c7ccfdf9afa@sentex.net> Date: Wed, 15 Jan 2020 10:14:31 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 47yW883tBvz4bsf X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.68 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.967,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.71)[ipnet: 2607:f3e0::/32(-4.93), asn: 11647(-3.54), country: CA(-0.09)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jan 2020 15:14:33 -0000 On 1/15/2020 9:55 AM, John Jasen wrote: > Executive summary: > > Periodically, load will spike on network interrupts on one of our > firewalls. Latency will quickly climb to the point that things are > unresponsive, sessions will timeout, and bandwidth will plummet. A couple of wild stabs... Are the routers generating any odd amount of ICMP response traffic at the time ? e.g. port|host unreachable etc ? (maybe track netstat -s -p icmp). Are there any bursts of icmp redirects happening ? I know that can slog a router sometimes-- Try instrumenting the appropriate oids (sysctl -a | grep -i redirect)  to see if thats the case.  A lot of small packets ?  If possible maybe a network tap in front of the boxes to capture / profile the traffic before/after to see if there is something like a big scan happening or DOS with many small packets etc.  If thats not possible, do you have enough spare CPU to do some netflow analysis on the box ? Or maybe take some periodic snapshots of the interface stats and compare normal to bad periods via sysctl -A dev.cxl | grep "_frames_" Good luck!     ---Mike