From owner-freebsd-net@freebsd.org Wed Jan 15 14:55:54 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9DB4F1F5841 for ; Wed, 15 Jan 2020 14:55:54 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: from mail-ua1-x92e.google.com (mail-ua1-x92e.google.com [IPv6:2607:f8b0:4864:20::92e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47yVkd50Nhz4Zxw for ; Wed, 15 Jan 2020 14:55:53 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: by mail-ua1-x92e.google.com with SMTP id y23so6335386ual.2 for ; Wed, 15 Jan 2020 06:55:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=rmWq/55NnaGZ8bQvR5ULB53S+kKJDi/vhwnQPel7KSo=; b=QYAE8aq0nCZvcjGPnP+pXd8pweGmHRFatruUkA7rW18VMWJ3fWu3Mxslsnrisge8Ln sVErsWZhlsABiSqwDVVtYYq0SNKhU1Si46ic1k2373z7mquGFqPtnGG4tRlgiGsEm1M7 1rUeF/QXUMyX5/E2CVdwYQ4EIxpKZ7/cPkX4iUxbJVec9Y0jvfmRgKXwsEpkztBDJPK+ J5npnGhqKguusUNq8LJkitucZq+YudVjBUBjgItY1mSw4IYdOzDzH/3LtugBQG4rYyHT YIrEMAWUbr3T1cJVLzG+0PKraXRdhnXEfM+anWxhBDawF0Llv/73GU+Nmn/bB8rL9PtW XDhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=rmWq/55NnaGZ8bQvR5ULB53S+kKJDi/vhwnQPel7KSo=; b=dWy56BhLm/A7pTrnG/+RJQ8sTT015ulxWRHvYJ/yvoOx6Yssa/+3oh0/1HVNl6TlUS OhDvme4OjHFvehQ78chqpLotjP5wwrKl17eSwnaeH7WKcT62Jz4NtaZPqo+MU3Xjbc7t 7SBCuUt2oHsDFH80ZVACri+Xgp2kAWs7Jlm1UBY+QpsC+iNm5WwJho6u5nPFXtrspYYj FzzBFwG9kBtRmQtV44mI0T2qi3U726z7Vzv5VKwnhGXxiJCz0kv54EamMyIBanxqBQ6R 4gd+ulvHpPn7F03p6nUUgmXyL/rmYW3iAbFLKWnzhOcVrx+3YSl5Jo4N27Ij7OHYskDQ SNFw== X-Gm-Message-State: APjAAAUcVhLswjUsKFDWgc4xLBz5UhOs/l6NJhGuZons/bNMOdlil61E pQ0xa3Rtl7VG2IBvdxbF2C5nwlB/Qx/r4QxlfY1qDSJ8 X-Google-Smtp-Source: APXvYqyp93SMkaJducN2QboSemTwt5jKOuVRlGAr+yTdMwjb3rvJJaLSAeKTSLZ0JKuis+JylHaArxqXO4H5iACcIhg= X-Received: by 2002:ab0:6258:: with SMTP id p24mr5635424uao.24.1579100152106; Wed, 15 Jan 2020 06:55:52 -0800 (PST) MIME-Version: 1.0 From: John Jasen Date: Wed, 15 Jan 2020 09:55:41 -0500 Message-ID: Subject: unexplained latency, interrupt spikes and loss of throughput on FreeBSD router/firewall system To: FreeBSD Net X-Rspamd-Queue-Id: 47yVkd50Nhz4Zxw X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=QYAE8aq0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jjasen@gmail.com designates 2607:f8b0:4864:20::92e as permitted sender) smtp.mailfrom=jjasen@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(0.00)[ip: (-9.41), ipnet: 2607:f8b0::/32(-2.09), asn: 15169(-1.83), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[e.2.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jan 2020 14:55:54 -0000 Executive summary: Periodically, load will spike on network interrupts on one of our firewalls. Latency will quickly climb to the point that things are unresponsive, sessions will timeout, and bandwidth will plummet. We do not see increases in ethernet pause frames, drops, errors, or anything else like that from the system. Usually, the quickest fix is to failover to the backup firewall. At that time, the backup firewall behaves normally and interrupt load drops on the afflicted firewall device. I'm stumped. Networking says its these systems. I believe its something on other side. Any ideas? Background information: FreeBSD 11.3-RELEASE-p3 hw.machine: amd64 hw.model: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz hw.ncpu: 24 hw.machine_arch: amd64 Firewall: pf failover: CARP network cards: seen with Chelsio T5-580 and T6 series cards. other networking information: VLANs are in use. Occasional LAGG usage as well. When this occurs, some of the interrupts dedicated to cxgbe queues spike to 100%. Latency climbs to the point that TCP timeouts start kicking in, and users start complaining. Bandwidth drops from 2-3Gbs to ~100-200Mbs netstat shows no increase of error or drop packets. sysctl shows no increase in pause frames. I'm happy to provide further information.