From owner-freebsd-net@freebsd.org Mon Jun 8 11:51:43 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C32AE32D124 for ; Mon, 8 Jun 2020 11:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 49gWnC4t6Vz4KNC for ; Mon, 8 Jun 2020 11:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id A752632D332; Mon, 8 Jun 2020 11:51:43 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A71A532D123 for ; Mon, 8 Jun 2020 11:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49gWnC3zDmz4KfJ for ; Mon, 8 Jun 2020 11:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 69B7B23E98 for ; Mon, 8 Jun 2020 11:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 058BphAM029974 for ; Mon, 8 Jun 2020 11:51:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 058BphKb029973 for net@FreeBSD.org; Mon, 8 Jun 2020 11:51:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 246951] carp(4): Active CARP member crashes: panic, trap_pfault, ip_input || ip_output when using ipSec, AES-NI (on Intel I350) Date: Mon, 08 Jun 2020 11:51:36 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.3-STABLE X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd-bugzilla@biscuit.ninja X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: mfc-stable12? mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2020 11:51:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246951 --- Comment #9 from freebsd-bugzilla@biscuit.ninja --- (In reply to Kubilay Kocak from comment #4) Thank you. I've attached dmesg.boot output. This is pfSense so there is no rc.conf. I've attached instead: - ifconfig output - interfaces and LAGGs specified in pfSense config.xml - sysctl tunables specified in pfSense config.xml The uptime between crashes varies between 2 and 30 days. It does not seem to correlate to any specific event that we are aware of or even peak throughpu= t. The only additional package installed on these firewalls is NRPE. In terms of workload: - HTTP/s traffic too and from customers - TCP load blancing of customer HTTP/s with 10 pools, 4 virtual servers per pool. Total of around 1.5 - 2 million active sessions - ipSec site-to-site tunnel for replication to our standby data centre - CARP / pfSync with bandwith/packet rates of 22-80 Mb/s, 2-8 Kpps - AES-NI enabled for IpSec (AES256-GCM) The firewalls are handling: - 20-45 Mb/s (13-45 Kpps) inbound ipSec - 30-150 Mb/s (14-55 Kpps) outbound ipSec - 20-90 Mb/s (15-60 Kpps) inbound IP traffic - 50-250 Mb/s (15-60 Kpps) outbound IP traffic - 30-90k states - ~66k Mbuf Clusters utilised (out of 1M total) The only other thing of note, that I can think of, is that we have a Cassan= dra cluster replicating over the IpSec tunnel. That's around 256 constantly changing states as data is replicated from one data centre to another. We have now disabled IpSec and switched to OpenVPN for the site-to-site VPN= , in order to see whether the crash is reproducable without IpSec Additionally, I had setup a couple of FreeBSD 11.3 VMs with a site-to-site IpSec connection. I had continuous iperf running over the tunnel for 7 days without issue. If there is any further information that I can provide, or anything I can d= o to assist, please don't hesitate. --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.=