From owner-freebsd-stable@freebsd.org Sun Nov 22 13:37:51 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6EBE468B32; Sun, 22 Nov 2020 13:37:51 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CfBDb0f8bz3Fr8; Sun, 22 Nov 2020 13:37:50 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:fb:4f0f:4a01:384c:e785:7e69:e5ee] (p200300Fb4f0F4A01384ce7857e69E5EE.dip0.t-ipconnect.de [IPv6:2003:fb:4f0f:4a01:384c:e785:7e69:e5ee]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 4CfBDN3cGJzFdH; Sun, 22 Nov 2020 14:37:40 +0100 (CET) From: Michael Grimm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: 12.2-STABLE: Commit 367740 breaks IMAP/SMTP server authentication Message-Id: Date: Sun, 22 Nov 2020 14:37:33 +0100 Cc: gnn@freebsd.org To: freebsd-net@freebsd.org, FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CfBDb0f8bz3Fr8 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of trashcan@ellael.org has no SPF policy when checking 91.121.41.56) smtp.mailfrom=trashcan@ellael.org X-Spamd-Result: default: False [-1.41 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[ellael.org]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[91.121.41.56:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[91.121.41.56:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.81)[-0.813]; AUTH_NA(1.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net,freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2020 13:37:51 -0000 Hi, I am running 12.2-STABLE and VNET jails, one of which host a recent = Dovecot IMAP and a recent postfix SMTP server. Authentication is forced = via TLS/SSL for both services (ports 587 and 993). Setup is as follows: extIF0/pf/NAT <=E2=80=94> epairXa (bridge0) epairXb <-> jail A recent upgrade broke mailing of IMAP clients running at macOS 10.14.6 = (Mojave) und AVM's push service (Fritzbox), but *not* for IMAP clients = running at macOS 10.15.7 (Catalina). Strange. Findings at macOS 10.14.6 (examplified for IMAP): 1) mac$ nc -4vw 1 mail.xyz.zzz 993 found 0 associations found 1 connections: 1: flags=3D82 outif en0 src 1.2.3.4 port 49583 dst 11.22.33.44 port 993 rank info not available TCP aux info available Connection to mail.xyz.zzz port 993 [tcp/imaps] succeeded! 2) mac$ openssl s_client -crlf -connect mail.xyz.zzz:993 -debug CONNECTED(00000005) write to 0x7fa32ef01ae0 [0x7fa33080a803] (200 bytes =3D> 200 = (0xC8)) 0000 - 16 03 01 00 c3 01 00 00-bf 03 03 32 f7 fe fa b4 = ...........2....=20 0010 - e8 9a 60 38 ef 34 99 70-84 ce dc 1a 08 b8 76 90 = ..`8.4.p=E2=80=A6=E2=80=A6v. 0020 - 19 8c 81 f4 a6 37 19 37-09 70 6f 00 00 60 c0 30 = .....7.7.po..`.0 0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 9f 00 6b 00 39 = .,.(.$.......k.9 0040 - cc a9 cc a8 cc aa ff 85-00 c4 00 88 00 81 00 9d = =E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6. 0050 - 00 3d 00 35 00 c0 00 84-c0 2f c0 2b c0 27 c0 23 = .=3D.5...../.+.'.# 0060 - c0 13 c0 09 00 9e 00 67-00 33 00 be 00 45 00 9c = .......g.3...E.. 0070 - 00 3c 00 2f 00 ba 00 41-c0 11 c0 07 00 05 00 04 = .<./...A=E2=80=A6=E2=80=A6.. 0080 - c0 12 c0 08 00 16 00 0a-00 15 00 09 00 ff 01 00 = =E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6. 0090 - 00 36 00 0b 00 02 01 00-00 0a 00 08 00 06 00 1d = .6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6.. 00a0 - 00 17 00 18 00 23 00 00-00 0d 00 1c 00 1a 06 01 = .....#=E2=80=A6=E2=80=A6=E2=80=A6. 00b0 - 06 03 ef ef 05 01 05 03-04 01 04 03 ee ee ed ed = =E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6. 00c0 - 03 01 03 03 02 01 02 03- = ........ hanging at that stage forever=20 (and client complaining of its inability to authenticate and = reports timeout after 60 seconds) I did identify commit 367740 being responsible for that: mike> svn up -r 367740 Updating '.': U sys/netinet/ip_fastfwd.c U sys/netinet/ip_input.c U sys/netinet/ip_var.h U . Updated to revision 367740. Any Ideas, especially why clients at different OS behave different? FYI: I do have no access to AVM's push service, and very limited access = to the macOS 10.14.6 computer. Thanks in advance and with kind regards, Michael P.S. How may I update a local svn copy and simultaneously omit commit = 367740 from being applied, or how may I revert commit 367740, only? From owner-freebsd-stable@freebsd.org Sun Nov 22 18:48:16 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0043A470CCB; Sun, 22 Nov 2020 18:48:16 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CfK6l66Dmz3qPH; Sun, 22 Nov 2020 18:48:15 +0000 (UTC) (envelope-from ronald-lists@klop.ws) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=klop.ws; s=mail; h=In-Reply-To:Message-ID:From:Content-Transfer-Encoding:MIME-Version: Date:References:Subject:Cc:To:Content-Type:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=F4Ab/7fTzJ9rdtDAIKB/70pILcE5wY8vdcIRoMkbk0E=; b=nhYz+Z2GeyhP3Du1rXck/gDi7e PTHfcqnn2yuJEwwvtPohlJO5RWc3pUOZm/fo5IIhswaLpiVGLV/cImQMV6RzKAK/MWWmwHFQUoFRN i7lUwneTwGU2AZF0e2YrM7HHVD2NVqE+CBrLqmO5TjgztoaAArdL8A7uFoNHVMuz1iYI=; Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-net@freebsd.org, "FreeBSD-STABLE Mailing List" , "Michael Grimm" Cc: gnn@freebsd.org Subject: Re: 12.2-STABLE: Commit 367740 breaks IMAP/SMTP server authentication References: Date: Sun, 22 Nov 2020 19:31:41 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.greenhost.nl X-Spam-Level: --- X-Spam-Score: -3.1 X-Spam-Status: No, score=-3.1 required=5.0 tests=ALL_TRUSTED, BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF autolearn=disabled version=3.4.2 X-Scan-Signature: 0420c3252dbee8b9b50dacb1a1909624 X-Rspamd-Queue-Id: 4CfK6l66Dmz3qPH X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2020 18:48:16 -0000 On Sun, 22 Nov 2020 14:37:33 +0100, Michael Grimm = = wrote: > Hi, > > I am running 12.2-STABLE and VNET jails, one of which host a recent = > Dovecot IMAP and a recent postfix SMTP server. Authentication is force= d = > via TLS/SSL for both services (ports 587 and 993). Setup is as follows= : > > extIF0/pf/NAT <=E2=80=94> epairXa (bridge0) epairXb <-> jail > > A recent upgrade broke mailing of IMAP clients running at macOS 10.14.= 6 = > (Mojave) und AVM's push service (Fritzbox), but *not* for IMAP clients= = > running at macOS 10.15.7 (Catalina). Strange. > > Findings at macOS 10.14.6 (examplified for IMAP): > > 1) mac$ nc -4vw 1 mail.xyz.zzz 993 > found 0 associations > found 1 connections: > 1: flags=3D82 > outif en0 > src 1.2.3.4 port 49583 > dst 11.22.33.44 port 993 > rank info not available > TCP aux info available > > Connection to mail.xyz.zzz port 993 [tcp/imaps] succeeded! > > 2) mac$ openssl s_client -crlf -connect mail.xyz.zzz:993 -debug > CONNECTED(00000005) > write to 0x7fa32ef01ae0 [0x7fa33080a803] (200 bytes =3D> 200 (0xC8)) > 0000 - 16 03 01 00 c3 01 00 00-bf 03 03 32 f7 fe fa b4 ...........2..= .. > 0010 - e8 9a 60 38 ef 34 99 70-84 ce dc 1a 08 b8 76 90 ..`8.4.p=E2=80= =A6=E2=80=A6v. > 0020 - 19 8c 81 f4 a6 37 19 37-09 70 6f 00 00 60 c0 30 = > .....7.7.po..`.0 > 0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 9f 00 6b 00 39 = > .,.(.$.......k.9 > 0040 - cc a9 cc a8 cc aa ff 85-00 c4 00 88 00 81 00 9d =E2=80=A6=E2= =80=A6=E2=80=A6=E2=80=A6=E2=80=A6. > 0050 - 00 3d 00 35 00 c0 00 84-c0 2f c0 2b c0 27 c0 23 = > .=3D.5...../.+.'.# > 0060 - c0 13 c0 09 00 9e 00 67-00 33 00 be 00 45 00 9c = > .......g.3...E.. > 0070 - 00 3c 00 2f 00 ba 00 41-c0 11 c0 07 00 05 00 04 .<./...A=E2=80= =A6=E2=80=A6.. > 0080 - c0 12 c0 08 00 16 00 0a-00 15 00 09 00 ff 01 00 =E2=80=A6=E2= =80=A6=E2=80=A6=E2=80=A6=E2=80=A6. > 0090 - 00 36 00 0b 00 02 01 00-00 0a 00 08 00 06 00 1d .6=E2=80=A6=E2= =80=A6=E2=80=A6=E2=80=A6.. > 00a0 - 00 17 00 18 00 23 00 00-00 0d 00 1c 00 1a 06 01 .....#=E2=80= =A6=E2=80=A6=E2=80=A6. > 00b0 - 06 03 ef ef 05 01 05 03-04 01 04 03 ee ee ed ed =E2=80=A6=E2= =80=A6=E2=80=A6=E2=80=A6=E2=80=A6. > 00c0 - 03 01 03 03 02 01 02 03- ........ > > hanging at that stage forever > (and client complaining of its inability to authenticate and reports = = > timeout after 60 seconds) > > > I did identify commit 367740 being responsible for that: > > mike> svn up -r 367740 > Updating '.': > U sys/netinet/ip_fastfwd.c > U sys/netinet/ip_input.c > U sys/netinet/ip_var.h > U . > Updated to revision 367740. > > > Any Ideas, especially why clients at different OS behave different? > > FYI: I do have no access to AVM's push service, and very limited acces= s = > to the macOS 10.14.6 computer. > > Thanks in advance and with kind regards, > Michael > > P.S. How may I update a local svn copy and simultaneously omit commit = = > 367740 from being applied, or how may I revert commit 367740, only? From the top of my head you can do something like: Assuming your svn checkout is in /usr/src: cd /usr/src svn up svn diff -c -367740 | patch This will get the reverse of commit 367740 (because of the -) and patch = = the code with it. Regards, Ronald. From owner-freebsd-stable@freebsd.org Sun Nov 22 19:00:04 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0F5954711FD; Sun, 22 Nov 2020 19:00:04 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:302:1100::e:1def]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CfKNM146Qz3qvg; Sun, 22 Nov 2020 19:00:02 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:fb:4f0f:4a01:384c:e785:7e69:e5ee] (p200300fB4f0F4a01384cE7857e69e5EE.dip0.t-ipconnect.de [IPv6:2003:fb:4f0f:4a01:384c:e785:7e69:e5ee]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 4CfKN95GqYzFxj; Sun, 22 Nov 2020 19:59:53 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: 12.2-STABLE: Commit 367740 breaks IMAP/SMTP server authentication From: Michael Grimm In-Reply-To: Date: Sun, 22 Nov 2020 19:59:46 +0100 Cc: gnn@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <9EED2425-AD5D-4171-97C0-591100FC297D@ellael.org> References: To: freebsd-net@freebsd.org, FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CfKNM146Qz3qvg X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of trashcan@ellael.org has no SPF policy when checking 2001:41d0:302:1100::e:1def) smtp.mailfrom=trashcan@ellael.org X-Spamd-Result: default: False [-1.60 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[ellael.org]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[2001:41d0:302:1100::e:1def:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:41d0:302:1100::e:1def:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; AUTH_NA(1.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:2001:41d0::/32, country:FR]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net,freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2020 19:00:04 -0000 Ronald Klop wrote: > On Sun, 22 Nov 2020 14:37:33 +0100, Michael Grimm = wrote: >> P.S. How may I update a local svn copy and simultaneously omit commit = 367740 from being applied, or how may I revert commit 367740, only? >=20 >=20 > =46rom the top of my head you can do something like: >=20 > Assuming your svn checkout is in /usr/src: > cd /usr/src > svn up > svn diff -c -367740 | patch >=20 > This will get the reverse of commit 367740 (because of the -) and = patch the code with it. Thanks, someone else pointed me to: svn merge -c -367740 . Worked as expected. Well, now I am able to omit this commit, but I would love to know what = is going on, and why this commit may break 'authentication/certificate = exchange/what so ever' of IMAP and SMTP/submission clients running in a = VNET jail ... With kind regards, Michael= From owner-freebsd-stable@freebsd.org Sun Nov 22 19:08:12 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 33FA9471848; Sun, 22 Nov 2020 19:08:12 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CfKYl1y2qz3rVS; Sun, 22 Nov 2020 19:08:11 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:fb:4f0f:4a01:384c:e785:7e69:e5ee] (p200300fB4f0F4a01384cE7857e69e5EE.dip0.t-ipconnect.de [IPv6:2003:fb:4f0f:4a01:384c:e785:7e69:e5ee]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 4CfKYj3ht7zG6M; Sun, 22 Nov 2020 20:08:09 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: 12.2-STABLE: Commit 367740 breaks IMAP/SMTP server authentication From: Michael Grimm In-Reply-To: <9EED2425-AD5D-4171-97C0-591100FC297D@ellael.org> Date: Sun, 22 Nov 2020 20:08:02 +0100 Cc: gnn@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <9EED2425-AD5D-4171-97C0-591100FC297D@ellael.org> To: freebsd-net@freebsd.org, FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CfKYl1y2qz3rVS X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of trashcan@ellael.org has no SPF policy when checking 91.121.41.56) smtp.mailfrom=trashcan@ellael.org X-Spamd-Result: default: False [-1.60 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[ellael.org]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[91.121.41.56:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[91.121.41.56:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; AUTH_NA(1.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net,freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Nov 2020 19:08:12 -0000 Hi - Michael Grimm wrote: > Well, now I am able to omit this commit, but I would love to know what = is going on, and why this commit may break 'authentication/certificate = exchange/what so ever' of IMAP and SMTP/submission clients running in a = VNET jail ... It just came to my mind, that I had had a strange issue with my setup = almots three years ago: = https://lists.freebsd.org/pipermail/freebsd-net/2018-January/049528.html /boot/loader.conf: # needs to become turned off (LRO) in order to restore tcp = performance within VNET jails: hw.vtnet.lro_disable=3D"1" hw.vtnet.tso_disable=3D"1" That is FYI, only. I have no clue if that's related anyhow. Regards, Michael From owner-freebsd-stable@freebsd.org Mon Nov 23 10:52:55 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 381114A728C for ; Mon, 23 Nov 2020 10:52:55 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq5.tb.mail.iss.as9143.net (smtpq5.tb.mail.iss.as9143.net [212.54.42.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CfkWn6jtKz3hxJ; Mon, 23 Nov 2020 10:52:53 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.135] (helo=smtp11.tb.mail.iss.as9143.net) by smtpq5.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kh9Sl-0005bz-8f; Mon, 23 Nov 2020 11:52:51 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp11.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1kh9Sk-0001gq-TA; Mon, 23 Nov 2020 11:52:50 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 5DD4212D; Mon, 23 Nov 2020 11:53:33 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id anIvP1CAnGqA; Mon, 23 Nov 2020 11:53:30 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 356D512C; Mon, 23 Nov 2020 11:53:30 +0100 (CET) From: Peter Blok Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Commit 367705+367706 causes a pabic Date: Mon, 23 Nov 2020 11:52:51 +0100 In-Reply-To: <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> Cc: FreeBSD Stable To: Kristof Provost References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=RMMNo6u+ c=1 sm=1 tr=0 ts=5fbb9482 a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=6Q3WNqvRAAAA:8 a=6I5d2MoRAAAA:8 a=N92cuIEdDoX1FTyWoFYA:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=I8PBwKCn76L9oNdl0isp:22 a=IjZwj45LgO3ly-622nXo:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4CfkWn6jtKz3hxJ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.168 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RECEIVED_SPAMHAUS_PBL(0.00)[82.101.198.11:received]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.54.42.168:from]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RCVD_IN_DNSWL_LOW(-0.10)[212.54.42.168:from]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[bsd4all.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[212.54.42.168:from:127.0.2.255]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2020 10:52:55 -0000 --Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Kristof, With commit 367705+367706 and if_bridge statically linked. It crashes = while adding an epair of a jail. With commit 367705+367706 and if_bridge dynamically loaded there is a = crash at reboot #0 0xffffffff8069ddc5 at kdb_backtrace+0x65 #1 0xffffffff80652c8b at vpanic+0x17b #2 0xffffffff80652b03 at panic+0x43 #3 0xffffffff809c8951 at trap_fatal+0x391 #4 0xffffffff809c89af at trap_pfault+0x4f #5 0xffffffff809c7ff6 at trap+0x286 #6 0xffffffff809a1ec8 at calltrap+0x8 #7 0xffffffff8079f7ed at ip_input+0x63d #8 0xffffffff8077a07a at netisr_dispatch_src+0xca #9 0xffffffff8075a6f8 at ether_demux+0x138 #10 0xffffffff8075b9bb at ether_nh_input+0x33b #11 0xffffffff8077a07a at netisr_dispatch_src+0xca #12 0xffffffff8075ab1b at ether_input+0x4b #13 0xffffffff8077a80b at swi_net+0x12b #14 0xffffffff8061e10c at ithread_loop+0x23c #15 0xffffffff8061afbe at fork_exit+0x7e #16 0xffffffff809a2efe at fork_trampoline+0xe Peter > On 21 Nov 2020, at 17:22, Peter Blok wrote: >=20 > Kristof, >=20 > With a GENERIC kernel it does NOT happen. I do have a different iflib = related panic at reboot, but I=E2=80=99ll report that separately. >=20 > I brought the two config files closer together and found out that if I = remove if_bridge from the config file and have it loaded dynamically = when the bridge is created, the problem no longer happens and everything = works ok. >=20 > Peter >=20 >> On 20 Nov 2020, at 15:53, Kristof Provost wrote: >>=20 >> I still can=E2=80=99t reproduce that panic. >>=20 >> Does it happen immediately after you start a vnet jail? >>=20 >> Does it also happen with a GENERIC kernel? >>=20 >> Regards, >> Kristof >>=20 >> On 20 Nov 2020, at 14:53, Peter Blok wrote: >>=20 >>> The panic with ipsec code in the backtrace was already very strange. = I was using IPsec, but only on one interface totally separate from the = members of the bridge as well as the bridge itself. The jails were not = doing any ipsec as well. Note that panic was a while ago and it was = after the 1st bridge epochification was done on stable-12 which was = later backed out >>>=20 >>> Today the system is no longer using ipsec, but it is still compiled = in. I can remove it if need be for a test >>>=20 >>>=20 >>> src.conf >>> WITHOUT_KERBEROS=3Dyes >>> WITHOUT_GSSAPI=3Dyes >>> WITHOUT_SENDMAIL=3Dtrue >>> WITHOUT_MAILWRAPPER=3Dtrue >>> WITHOUT_DMAGENT=3Dtrue >>> WITHOUT_GAMES=3Dtrue >>> WITHOUT_IPFILTER=3Dtrue >>> WITHOUT_UNBOUND=3Dtrue >>> WITHOUT_PROFILE=3Dtrue >>> WITHOUT_ATM=3Dtrue >>> WITHOUT_BSNMP=3Dtrue >>> #WITHOUT_CROSS_COMPILER=3Dtrue >>> WITHOUT_DEBUG_FILES=3Dtrue >>> WITHOUT_DICT=3Dtrue >>> WITHOUT_FLOPPY=3Dtrue >>> WITHOUT_HTML=3Dtrue >>> WITHOUT_HYPERV=3Dtrue >>> WITHOUT_NDIS=3Dtrue >>> WITHOUT_NIS=3Dtrue >>> WITHOUT_PPP=3Dtrue >>> WITHOUT_TALK=3Dtrue >>> WITHOUT_TESTS=3Dtrue >>> WITHOUT_WIRELESS=3Dtrue >>> #WITHOUT_LIB32=3Dtrue >>> WITHOUT_LPR=3Dtrue >>>=20 >>> make.conf >>> KERNCONF=3DBHYVE >>> MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge = bridgestp if_vxlan pflog libmchain libiconv smbfs linux linux64 = linux_common linuxkpi linprocfs linsysfs ext2fs >>> DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 = python3=3D3.8 >>> OPTIONS_UNSET=3DDOCS NLS MANPAGES >>>=20 >>> BHYVE >>> cpu HAMMER >>> ident BHYVE >>>=20 >>> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug = symbols >>> makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support >>>=20 >>> options CAMDEBUG >>>=20 >>> options SCHED_ULE # ULE scheduler >>> options PREEMPTION # Enable kernel thread = preemption >>> options INET # InterNETworking >>> options INET6 # IPv6 communications protocols >>> options IPSEC >>> options TCP_OFFLOAD # TCP offload >>> options TCP_RFC7413 # TCP FASTOPEN >>> options SCTP # Stream Control Transmission = Protocol >>> options FFS # Berkeley Fast Filesystem >>> options SOFTUPDATES # Enable FFS soft updates = support >>> options UFS_ACL # Support for access control = lists >>> options UFS_DIRHASH # Improve performance on big = directories >>> options UFS_GJOURNAL # Enable gjournal-based UFS = journaling >>> options QUOTA # Enable disk quotas for UFS >>> options SUIDDIR >>> options NFSCL # Network Filesystem Client >>> options NFSD # Network Filesystem Server >>> options NFSLOCKD # Network Lock Manager >>> options MSDOSFS # MSDOS Filesystem >>> options CD9660 # ISO 9660 Filesystem >>> options FUSEFS >>> options NULLFS # NULL filesystem >>> options UNIONFS >>> options FDESCFS # File descriptor = filesystem >>> options PROCFS # Process filesystem (requires = PSEUDOFS) >>> options PSEUDOFS # Pseudo-filesystem framework >>> options GEOM_PART_GPT # GUID Partition Tables. >>> options GEOM_RAID # Soft RAID functionality. >>> options GEOM_LABEL # Provides labelization >>> options GEOM_ELI # Disk encryption. >>> options COMPAT_FREEBSD32 # Compatible with i386 binaries >>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >>> options COMPAT_FREEBSD9 # Compatible with FreeBSD9 >>> options COMPAT_FREEBSD10 # Compatible with FreeBSD10 >>> options COMPAT_FREEBSD11 # Compatible with FreeBSD11 >>> options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI >>> options KTRACE # ktrace(1) support >>> options STACK # stack(9) support >>> options SYSVSHM # SYSV-style shared memory >>> options SYSVMSG # SYSV-style message queues >>> options SYSVSEM # SYSV-style semaphores >>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions >>> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. >>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >>> options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) >>> options AUDIT # Security event auditing >>> options CAPABILITY_MODE # Capsicum capability mode >>> options CAPABILITIES # Capsicum capabilities >>> options MAC # TrustedBSD MAC Framework >>> options MAC_PORTACL >>> options MAC_NTPD >>> options KDTRACE_FRAME # Ensure frames are compiled in >>> options KDTRACE_HOOKS # Kernel DTrace hooks >>> options DDB_CTF # Kernel ELF linker loads CTF = data >>> options INCLUDE_CONFIG_FILE # Include this file in kernel >>>=20 >>> # Debugging support. Always need this: >>> options KDB # Enable kernel debugger = support. >>> options KDB_TRACE # Print a stack trace for a = panic. >>> options KDB_UNATTENDED >>>=20 >>> # Make an SMP-capable kernel by default >>> options SMP # Symmetric MultiProcessor = Kernel >>> options EARLY_AP_STARTUP >>>=20 >>> # CPU frequency control >>> device cpufreq >>> device cpuctl >>> device coretemp >>>=20 >>> # Bus support. >>> device acpi >>> options ACPI_DMAR >>> device pci >>> options PCI_IOV # PCI SR-IOV support >>>=20 >>> device iicbus >>> device iicbb >>>=20 >>> device iic >>> device ic >>> device iicsmb >>>=20 >>> device ichsmb >>> device smbus >>> device smb >>>=20 >>> #device jedec_dimm >>>=20 >>> # ATA controllers >>> device ahci # AHCI-compatible SATA = controllers >>> device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA >>>=20 >>> # SCSI Controllers >>> device mps # LSI-Logic MPT-Fusion 2 >>>=20 >>> # ATA/SCSI peripherals >>> device scbus # SCSI bus (required for = ATA/SCSI) >>> device da # Direct Access (disks) >>> device cd # CD >>> device pass # Passthrough device = (direct ATA/SCSI access) >>> device ses # Enclosure Services = (SES and SAF-TE) >>> device sg >>>=20 >>> device cfiscsi >>> device ctl # CAM Target Layer >>> device iscsi >>>=20 >>> # atkbdc0 controls both the keyboard and the PS/2 mouse >>> device atkbdc # AT keyboard controller >>> device atkbd # AT keyboard >>> device psm # PS/2 mouse >>>=20 >>> device kbdmux # keyboard multiplexer >>>=20 >>> # vt is the new video console driver >>> device vt >>> device vt_vga >>> device vt_efifb >>>=20 >>> # Serial (COM) ports >>> device uart # Generic UART driver >>>=20 >>> # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure >>> device iflib >>> device em # Intel PRO/1000 Gigabit = Ethernet Family >>> device ix # Intel PRO/10GbE PCIE = PF Ethernet >>>=20 >>> # Network stack virtualization. >>> options VIMAGE >>>=20 >>> # Pseudo devices. >>> device crypto >>> device cryptodev >>> device loop # Network loopback >>> device random # Entropy device >>> device padlock_rng # VIA Padlock RNG >>> device rdrand_rng # Intel Bull Mountain = RNG >>> device ipmi >>> device smbios >>> device vpd >>> device aesni # AES-NI OpenCrypto = module >>> device ether # Ethernet support >>> device lagg >>> device vlan # 802.1Q VLAN support >>> device tuntap # Packet tunnel. >>> device md # Memory "disks" >>> device gif # IPv6 and IPv4 = tunneling >>> device firmware # firmware assist module >>>=20 >>> device pf >>> #device pflog >>> #device pfsync >>>=20 >>> # The `bpf' device enables the Berkeley Packet Filter. >>> # Be aware of the administrative consequences of enabling this! >>> # Note that 'bpf' is required for DHCP. >>> device bpf # Berkeley packet filter >>>=20 >>> # The `epair' device implements a virtual back-to-back connected = Ethernet >>> # like interface pair. >>> device epair >>>=20 >>> # USB support >>> options USB_DEBUG # enable debug msgs >>> device uhci # UHCI PCI->USB = interface >>> device ohci # OHCI PCI->USB = interface >>> device ehci # EHCI PCI->USB = interface (USB 2.0) >>> device xhci # XHCI PCI->USB = interface (USB 3.0) >>> device usb # USB Bus (required) >>> device uhid >>> device ukbd # Keyboard >>> device umass # Disks/Mass storage - = Requires scbus and da >>> device ums >>>=20 >>> device filemon >>>=20 >>> device if_bridge >>>=20 >>>> On 20 Nov 2020, at 12:53, Kristof Provost wrote: >>>>=20 >>>> Can you share your kernel config file (and src.conf / make.conf if = they exist)? >>>>=20 >>>> This second panic is in the IPSec code. My current thinking is that = your kernel config is triggering a bug that=E2=80=99s manifesting in = multiple places, but not actually caused by those places. >>>>=20 >>>> I=E2=80=99d like to be able to reproduce it so we can debug it. >>>>=20 >>>> Best regards, >>>> Kristof >>>>=20 >>>> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>>>> Hi Kristof, >>>>>=20 >>>>> This is 12-stable. With the previous bridge epochification that = was backed out my config had a panic too. >>>>>=20 >>>>> I don=E2=80=99t have any local modifications. I did a clean = rebuild after removing /usr/obj/usr >>>>>=20 >>>>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko = and nmdm.ko as modules. Everything else is statically linked. I have = removed all drivers not needed for the hardware at hand. >>>>>=20 >>>>> My bridge is between two vlans from the same trunk and the jail = epair devices as well as the bhyve tap devices. >>>>>=20 >>>>> The panic happens when the jails are starting. >>>>>=20 >>>>> I can try to narrow it down over the weekend and make the crash = dump available for analysis. >>>>>=20 >>>>> Previously I had the following crash with 363492 >>>>>=20 >>>>> kernel trap 12 with interrupts disabled >>>>>=20 >>>>>=20 >>>>> Fatal trap 12: page fault while in kernel mode >>>>> cpuid =3D 2; apic id =3D 02 >>>>> fault virtual address =3D 0xffffffff00000410 >>>>> fault code =3D supervisor read data, page not = present >>>>> instruction pointer =3D 0x20:0xffffffff80692326 >>>>> stack pointer =3D 0x28:0xfffffe00c06097b0 >>>>> frame pointer =3D 0x28:0xfffffe00c06097f0 >>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>>> processor eflags =3D resume, IOPL =3D 0 >>>>> current process =3D 2030 (ifconfig) >>>>> trap number =3D 12 >>>>> panic: page fault >>>>> cpuid =3D 2 >>>>> time =3D 1595683412 >>>>> KDB: stack backtrace: >>>>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>>>> #1 0xffffffff8064d67b at vpanic+0x17b >>>>> #2 0xffffffff8064d4f3 at panic+0x43 >>>>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>>>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>>>> #5 0xffffffff809cb9b6 at trap+0x286 >>>>> #6 0xffffffff809a5b28 at calltrap+0x8 >>>>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>>>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>>>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>>>> #10 0xffffffff8075274f at ifioctl+0x47f >>>>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>>>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>>>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>>>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On 20 Nov 2020, at 11:30, Kristof Provost wrote: >>>>>>=20 >>>>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = wrote: >>>>>>> I=E2=80=99m afraid the last Epoch fix for bridge is not solving = the problem ( or perhaps creates a new ). >>>>>>>=20 >>>>>> We=E2=80=99re talking about the stable/12 branch, right? >>>>>>=20 >>>>>>> This seems to happen when the jail epair is added to the bridge. >>>>>>>=20 >>>>>> There must be something more to it than that. I=E2=80=99ve run = the bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>>>>=20 >>>>>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>>>>=20 >>>>>>> kernel trap 12 with interrupts disabled >>>>>>>=20 >>>>>>>=20 >>>>>>> Fatal trap 12: page fault while in kernel mode >>>>>>> cpuid =3D 6; apic id =3D 06 >>>>>>> fault virtual address =3D 0xc10 >>>>>>> fault code =3D supervisor read data, page not = present >>>>>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>>>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>>>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>>>> =3D DPL 0, pres 1, long 1, def32 0, gran = 1 >>>>>>> processor eflags =3D resume, IOPL =3D 0 >>>>>>> current process =3D 1686 (jail) >>>>>>> trap number =3D 12 >>>>>>> panic: page fault >>>>>>> cpuid =3D 6 >>>>>>> time =3D 1605811310 >>>>>>> KDB: stack backtrace: >>>>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>>>>=20 >>>>>> This panic is rather odd. This isn=E2=80=99t even the bridge = code. This is during initial creation of the vnet. I don=E2=80=99t = really see how this could even trigger panics. >>>>>> That panic looks as if something corrupted the net_epoch_preempt, = by overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >>>>>>=20 >>>>>> Best regards, >>>>>> Kristof >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 --Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIzMTA1MjUxWjAvBgkqhkiG9w0BCQQxIgQgylwhoXHE RB0XAzFzJW/cnRjL+y0Qj5NdIuGXjLpIaaYwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBABf1fGOn9oO4fqG7NMUn8YzqEduGArgd RAIGoPj+QDUXPX0e+DSMWHA7k1ZDjfKX5rhN92qIZVAMG/Egea010SjNoERAmx6oM5qcTy70l0eQ BKMyNfBYSIxWqRBEhIZW2z2pvDLy46bEh5vaKC0cj7WjkUUt1qs6P/PeR3oAEz9MZNHfFo7u9Et9 ABW3xSEMPmFtTIG6EyBlr6jPXAbZ/ymLy6shxlh4yTnjDHYcJlm1b/Z7orJV4ud67cjj00b14h2c WYGV2pIYc+t40D45mmGPCV2yslKnLLbm/Db1VQP5SslTQE4KlTxvJHh38u3jC2jhnXdVoW/DEoIV H+ZJKQoAAAAAAAA= --Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4-- From owner-freebsd-stable@freebsd.org Mon Nov 23 11:15:54 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 114E34A7B0D for ; Mon, 23 Nov 2020 11:15:54 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cfl2K6S45z3jcZ; Mon, 23 Nov 2020 11:15:53 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 80790ED77; Mon, 23 Nov 2020 11:15:53 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 231907C66; Mon, 23 Nov 2020 12:15:51 +0100 (CET) From: "Kristof Provost" To: "Peter Blok" Cc: "FreeBSD Stable" Subject: Re: Commit 367705+367706 causes a pabic Date: Mon, 23 Nov 2020 12:15:50 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2020 11:15:54 -0000 Peter, Is that backtrace from the first or the second situation you describe? What kernel config are you using with that backtrace? This backtrace does not appear to involve the bridge. Given that part of the panic message is cut off it’s very hard to conclude anything at all from it. Best regards, Kristof On 23 Nov 2020, at 11:52, Peter Blok wrote: > Kristof, > > With commit 367705+367706 and if_bridge statically linked. It crashes > while adding an epair of a jail. > > With commit 367705+367706 and if_bridge dynamically loaded there is a > crash at reboot > > #0 0xffffffff8069ddc5 at kdb_backtrace+0x65 > #1 0xffffffff80652c8b at vpanic+0x17b > #2 0xffffffff80652b03 at panic+0x43 > #3 0xffffffff809c8951 at trap_fatal+0x391 > #4 0xffffffff809c89af at trap_pfault+0x4f > #5 0xffffffff809c7ff6 at trap+0x286 > #6 0xffffffff809a1ec8 at calltrap+0x8 > #7 0xffffffff8079f7ed at ip_input+0x63d > #8 0xffffffff8077a07a at netisr_dispatch_src+0xca > #9 0xffffffff8075a6f8 at ether_demux+0x138 > #10 0xffffffff8075b9bb at ether_nh_input+0x33b > #11 0xffffffff8077a07a at netisr_dispatch_src+0xca > #12 0xffffffff8075ab1b at ether_input+0x4b > #13 0xffffffff8077a80b at swi_net+0x12b > #14 0xffffffff8061e10c at ithread_loop+0x23c > #15 0xffffffff8061afbe at fork_exit+0x7e > #16 0xffffffff809a2efe at fork_trampoline+0xe > > Peter > >> On 21 Nov 2020, at 17:22, Peter Blok wrote: >> >> Kristof, >> >> With a GENERIC kernel it does NOT happen. I do have a different iflib >> related panic at reboot, but I’ll report that separately. >> >> I brought the two config files closer together and found out that if >> I remove if_bridge from the config file and have it loaded >> dynamically when the bridge is created, the problem no longer happens >> and everything works ok. >> >> Peter >> >>> On 20 Nov 2020, at 15:53, Kristof Provost wrote: >>> >>> I still can’t reproduce that panic. >>> >>> Does it happen immediately after you start a vnet jail? >>> >>> Does it also happen with a GENERIC kernel? >>> >>> Regards, >>> Kristof >>> >>> On 20 Nov 2020, at 14:53, Peter Blok wrote: >>> >>>> The panic with ipsec code in the backtrace was already very >>>> strange. I was using IPsec, but only on one interface totally >>>> separate from the members of the bridge as well as the bridge >>>> itself. The jails were not doing any ipsec as well. Note that panic >>>> was a while ago and it was after the 1st bridge epochification was >>>> done on stable-12 which was later backed out >>>> >>>> Today the system is no longer using ipsec, but it is still compiled >>>> in. I can remove it if need be for a test >>>> >>>> >>>> src.conf >>>> WITHOUT_KERBEROS=yes >>>> WITHOUT_GSSAPI=yes >>>> WITHOUT_SENDMAIL=true >>>> WITHOUT_MAILWRAPPER=true >>>> WITHOUT_DMAGENT=true >>>> WITHOUT_GAMES=true >>>> WITHOUT_IPFILTER=true >>>> WITHOUT_UNBOUND=true >>>> WITHOUT_PROFILE=true >>>> WITHOUT_ATM=true >>>> WITHOUT_BSNMP=true >>>> #WITHOUT_CROSS_COMPILER=true >>>> WITHOUT_DEBUG_FILES=true >>>> WITHOUT_DICT=true >>>> WITHOUT_FLOPPY=true >>>> WITHOUT_HTML=true >>>> WITHOUT_HYPERV=true >>>> WITHOUT_NDIS=true >>>> WITHOUT_NIS=true >>>> WITHOUT_PPP=true >>>> WITHOUT_TALK=true >>>> WITHOUT_TESTS=true >>>> WITHOUT_WIRELESS=true >>>> #WITHOUT_LIB32=true >>>> WITHOUT_LPR=true >>>> >>>> make.conf >>>> KERNCONF=BHYVE >>>> MODULES_OVERRIDE=opensolaris dtrace zfs vmm nmdm if_bridge >>>> bridgestp if_vxlan pflog libmchain libiconv smbfs linux linux64 >>>> linux_common linuxkpi linprocfs linsysfs ext2fs >>>> DEFAULT_VERSIONS+=perl5=5.30 mysql=5.7 python=3.8 python3=3.8 >>>> OPTIONS_UNSET=DOCS NLS MANPAGES >>>> >>>> BHYVE >>>> cpu HAMMER >>>> ident BHYVE >>>> >>>> makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols >>>> makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support >>>> >>>> options CAMDEBUG >>>> >>>> options SCHED_ULE # ULE scheduler >>>> options PREEMPTION # Enable kernel thread preemption >>>> options INET # InterNETworking >>>> options INET6 # IPv6 communications protocols >>>> options IPSEC >>>> options TCP_OFFLOAD # TCP offload >>>> options TCP_RFC7413 # TCP FASTOPEN >>>> options SCTP # Stream Control Transmission Protocol >>>> options FFS # Berkeley Fast Filesystem >>>> options SOFTUPDATES # Enable FFS soft updates support >>>> options UFS_ACL # Support for access control lists >>>> options UFS_DIRHASH # Improve performance on big directories >>>> options UFS_GJOURNAL # Enable gjournal-based UFS journaling >>>> options QUOTA # Enable disk quotas for UFS >>>> options SUIDDIR >>>> options NFSCL # Network Filesystem Client >>>> options NFSD # Network Filesystem Server >>>> options NFSLOCKD # Network Lock Manager >>>> options MSDOSFS # MSDOS Filesystem >>>> options CD9660 # ISO 9660 Filesystem >>>> options FUSEFS >>>> options NULLFS # NULL filesystem >>>> options UNIONFS >>>> options FDESCFS # File descriptor filesystem >>>> options PROCFS # Process filesystem (requires PSEUDOFS) >>>> options PSEUDOFS # Pseudo-filesystem framework >>>> options GEOM_PART_GPT # GUID Partition Tables. >>>> options GEOM_RAID # Soft RAID functionality. >>>> options GEOM_LABEL # Provides labelization >>>> options GEOM_ELI # Disk encryption. >>>> options COMPAT_FREEBSD32 # Compatible with i386 binaries >>>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >>>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >>>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >>>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >>>> options COMPAT_FREEBSD9 # Compatible with FreeBSD9 >>>> options COMPAT_FREEBSD10 # Compatible with FreeBSD10 >>>> options COMPAT_FREEBSD11 # Compatible with FreeBSD11 >>>> options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI >>>> options KTRACE # ktrace(1) support >>>> options STACK # stack(9) support >>>> options SYSVSHM # SYSV-style shared memory >>>> options SYSVMSG # SYSV-style message queues >>>> options SYSVSEM # SYSV-style semaphores >>>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time >>>> extensions >>>> options PRINTF_BUFR_SIZE=128 # Prevent printf output being >>>> interspersed. >>>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >>>> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) >>>> options AUDIT # Security event auditing >>>> options CAPABILITY_MODE # Capsicum capability mode >>>> options CAPABILITIES # Capsicum capabilities >>>> options MAC # TrustedBSD MAC Framework >>>> options MAC_PORTACL >>>> options MAC_NTPD >>>> options KDTRACE_FRAME # Ensure frames are compiled in >>>> options KDTRACE_HOOKS # Kernel DTrace hooks >>>> options DDB_CTF # Kernel ELF linker loads CTF data >>>> options INCLUDE_CONFIG_FILE # Include this file in kernel >>>> >>>> # Debugging support. Always need this: >>>> options KDB # Enable kernel debugger support. >>>> options KDB_TRACE # Print a stack trace for a panic. >>>> options KDB_UNATTENDED >>>> >>>> # Make an SMP-capable kernel by default >>>> options SMP # Symmetric MultiProcessor Kernel >>>> options EARLY_AP_STARTUP >>>> >>>> # CPU frequency control >>>> device cpufreq >>>> device cpuctl >>>> device coretemp >>>> >>>> # Bus support. >>>> device acpi >>>> options ACPI_DMAR >>>> device pci >>>> options PCI_IOV # PCI SR-IOV support >>>> >>>> device iicbus >>>> device iicbb >>>> >>>> device iic >>>> device ic >>>> device iicsmb >>>> >>>> device ichsmb >>>> device smbus >>>> device smb >>>> >>>> #device jedec_dimm >>>> >>>> # ATA controllers >>>> device ahci # AHCI-compatible SATA controllers >>>> device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA >>>> >>>> # SCSI Controllers >>>> device mps # LSI-Logic MPT-Fusion 2 >>>> >>>> # ATA/SCSI peripherals >>>> device scbus # SCSI bus (required for ATA/SCSI) >>>> device da # Direct Access (disks) >>>> device cd # CD >>>> device pass # Passthrough device (direct ATA/SCSI access) >>>> device ses # Enclosure Services (SES and SAF-TE) >>>> device sg >>>> >>>> device cfiscsi >>>> device ctl # CAM Target Layer >>>> device iscsi >>>> >>>> # atkbdc0 controls both the keyboard and the PS/2 mouse >>>> device atkbdc # AT keyboard controller >>>> device atkbd # AT keyboard >>>> device psm # PS/2 mouse >>>> >>>> device kbdmux # keyboard multiplexer >>>> >>>> # vt is the new video console driver >>>> device vt >>>> device vt_vga >>>> device vt_efifb >>>> >>>> # Serial (COM) ports >>>> device uart # Generic UART driver >>>> >>>> # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure >>>> device iflib >>>> device em # Intel PRO/1000 Gigabit Ethernet Family >>>> device ix # Intel PRO/10GbE PCIE PF Ethernet >>>> >>>> # Network stack virtualization. >>>> options VIMAGE >>>> >>>> # Pseudo devices. >>>> device crypto >>>> device cryptodev >>>> device loop # Network loopback >>>> device random # Entropy device >>>> device padlock_rng # VIA Padlock RNG >>>> device rdrand_rng # Intel Bull Mountain RNG >>>> device ipmi >>>> device smbios >>>> device vpd >>>> device aesni # AES-NI OpenCrypto module >>>> device ether # Ethernet support >>>> device lagg >>>> device vlan # 802.1Q VLAN support >>>> device tuntap # Packet tunnel. >>>> device md # Memory "disks" >>>> device gif # IPv6 and IPv4 tunneling >>>> device firmware # firmware assist module >>>> >>>> device pf >>>> #device pflog >>>> #device pfsync >>>> >>>> # The `bpf' device enables the Berkeley Packet Filter. >>>> # Be aware of the administrative consequences of enabling this! >>>> # Note that 'bpf' is required for DHCP. >>>> device bpf # Berkeley packet filter >>>> >>>> # The `epair' device implements a virtual back-to-back connected >>>> Ethernet >>>> # like interface pair. >>>> device epair >>>> >>>> # USB support >>>> options USB_DEBUG # enable debug msgs >>>> device uhci # UHCI PCI->USB interface >>>> device ohci # OHCI PCI->USB interface >>>> device ehci # EHCI PCI->USB interface (USB 2.0) >>>> device xhci # XHCI PCI->USB interface (USB 3.0) >>>> device usb # USB Bus (required) >>>> device uhid >>>> device ukbd # Keyboard >>>> device umass # Disks/Mass storage - Requires scbus and da >>>> device ums >>>> >>>> device filemon >>>> >>>> device if_bridge >>>> >>>>> On 20 Nov 2020, at 12:53, Kristof Provost wrote: >>>>> >>>>> Can you share your kernel config file (and src.conf / make.conf if >>>>> they exist)? >>>>> >>>>> This second panic is in the IPSec code. My current thinking is >>>>> that your kernel config is triggering a bug that’s manifesting >>>>> in multiple places, but not actually caused by those places. >>>>> >>>>> I’d like to be able to reproduce it so we can debug it. >>>>> >>>>> Best regards, >>>>> Kristof >>>>> >>>>> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>>>>> Hi Kristof, >>>>>> >>>>>> This is 12-stable. With the previous bridge epochification that >>>>>> was backed out my config had a panic too. >>>>>> >>>>>> I don’t have any local modifications. I did a clean rebuild >>>>>> after removing /usr/obj/usr >>>>>> >>>>>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko >>>>>> and nmdm.ko as modules. Everything else is statically linked. I >>>>>> have removed all drivers not needed for the hardware at hand. >>>>>> >>>>>> My bridge is between two vlans from the same trunk and the jail >>>>>> epair devices as well as the bhyve tap devices. >>>>>> >>>>>> The panic happens when the jails are starting. >>>>>> >>>>>> I can try to narrow it down over the weekend and make the crash >>>>>> dump available for analysis. >>>>>> >>>>>> Previously I had the following crash with 363492 >>>>>> >>>>>> kernel trap 12 with interrupts disabled >>>>>> >>>>>> >>>>>> Fatal trap 12: page fault while in kernel mode >>>>>> cpuid = 2; apic id = 02 >>>>>> fault virtual address = 0xffffffff00000410 >>>>>> fault code = supervisor read data, page not present >>>>>> instruction pointer = 0x20:0xffffffff80692326 >>>>>> stack pointer = 0x28:0xfffffe00c06097b0 >>>>>> frame pointer = 0x28:0xfffffe00c06097f0 >>>>>> code segment = base 0x0, limit 0xfffff, type 0x1b >>>>>> = DPL 0, pres 1, long 1, def32 0, gran 1 >>>>>> processor eflags = resume, IOPL = 0 >>>>>> current process = 2030 (ifconfig) >>>>>> trap number = 12 >>>>>> panic: page fault >>>>>> cpuid = 2 >>>>>> time = 1595683412 >>>>>> KDB: stack backtrace: >>>>>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>>>>> #1 0xffffffff8064d67b at vpanic+0x17b >>>>>> #2 0xffffffff8064d4f3 at panic+0x43 >>>>>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>>>>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>>>>> #5 0xffffffff809cb9b6 at trap+0x286 >>>>>> #6 0xffffffff809a5b28 at calltrap+0x8 >>>>>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>>>>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>>>>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>>>>> #10 0xffffffff8075274f at ifioctl+0x47f >>>>>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>>>>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>>>>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>>>>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On 20 Nov 2020, at 11:30, Kristof Provost >>>>>>> wrote: >>>>>>> >>>>>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org >>>>>>> wrote: >>>>>>>> I’m afraid the last Epoch fix for bridge is not solving the >>>>>>>> problem ( or perhaps creates a new ). >>>>>>>> >>>>>>> We’re talking about the stable/12 branch, right? >>>>>>> >>>>>>>> This seems to happen when the jail epair is added to the >>>>>>>> bridge. >>>>>>>> >>>>>>> There must be something more to it than that. I’ve run the >>>>>>> bridge tests on stable/12 without issue, and this is a problem >>>>>>> we didn’t see when the bridge epochification initially went >>>>>>> into stable/12. >>>>>>> >>>>>>> Do you have a custom kernel config? Other patches? What exact >>>>>>> commands do you run to trigger the panic? >>>>>>> >>>>>>>> kernel trap 12 with interrupts disabled >>>>>>>> >>>>>>>> >>>>>>>> Fatal trap 12: page fault while in kernel mode >>>>>>>> cpuid = 6; apic id = 06 >>>>>>>> fault virtual address = 0xc10 >>>>>>>> fault code = supervisor read data, page not present >>>>>>>> instruction pointer = 0x20:0xffffffff80695e76 >>>>>>>> stack pointer = 0x28:0xfffffe00bf14e6e0 >>>>>>>> frame pointer = 0x28:0xfffffe00bf14e720 >>>>>>>> code segment = base 0x0, limit 0xfffff, type 0x1b >>>>>>>> = DPL 0, pres 1, long 1, def32 0, gran 1 >>>>>>>> processor eflags = resume, IOPL = 0 >>>>>>>> current process = 1686 (jail) >>>>>>>> trap number = 12 >>>>>>>> panic: page fault >>>>>>>> cpuid = 6 >>>>>>>> time = 1605811310 >>>>>>>> KDB: stack backtrace: >>>>>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>>>>> >>>>>>> This panic is rather odd. This isn’t even the bridge code. >>>>>>> This is during initial creation of the vnet. I don’t really >>>>>>> see how this could even trigger panics. >>>>>>> That panic looks as if something corrupted the >>>>>>> net_epoch_preempt, by overwriting the epoch->e_epoch. The bridge >>>>>>> patches only access this variable through the well-established >>>>>>> functions and macros. I see no obvious way that they could >>>>>>> corrupt it. >>>>>>> >>>>>>> Best regards, >>>>>>> Kristof >>>>> >>>>> >>>>> _______________________________________________ >>>>> freebsd-stable@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>>> To unsubscribe, send any mail to >>>>> "freebsd-stable-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to >>> "freebsd-stable-unsubscribe@freebsd.org" >> From owner-freebsd-stable@freebsd.org Mon Nov 23 12:27:23 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 16AFA2EAD3D for ; Mon, 23 Nov 2020 12:27:23 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq5.tb.mail.iss.as9143.net (smtpq5.tb.mail.iss.as9143.net [212.54.42.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cfmcp6Dxmz3plp; Mon, 23 Nov 2020 12:27:22 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.136] (helo=smtp12.tb.mail.iss.as9143.net) by smtpq5.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1khAwC-00012c-KJ; Mon, 23 Nov 2020 13:27:20 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp12.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1khAwC-0003z4-8B; Mon, 23 Nov 2020 13:27:20 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 730921DC; Mon, 23 Nov 2020 13:28:02 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z10cwaKTEzvY; Mon, 23 Nov 2020 13:27:58 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 6D20832D; Mon, 23 Nov 2020 13:27:58 +0100 (CET) From: Peter Blok Message-Id: <8142230B-C47E-447F-9EF1-5DB46AA1C7DC@bsd4all.org> Content-Type: multipart/signed; boundary="Apple-Mail=_2AEED608-5094-46C3-AD01-832A8DF68E09"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Commit 367705+367706 causes a pabic Date: Mon, 23 Nov 2020 13:27:19 +0100 In-Reply-To: Cc: FreeBSD Stable To: Kristof Provost References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=fshi2H0f c=1 sm=1 tr=0 ts=5fbbaaa8 a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=6I5d2MoRAAAA:8 a=6Q3WNqvRAAAA:8 a=fHBQ5mLlmN0NXt4SzcYA:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=IjZwj45LgO3ly-622nXo:22 a=I8PBwKCn76L9oNdl0isp:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4Cfmcp6Dxmz3plp X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2020 12:27:23 -0000 --Apple-Mail=_2AEED608-5094-46C3-AD01-832A8DF68E09 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Kristof, It=E2=80=99s from the 2nd situation. It is so weird. Last time there was = ipsec code in the backtrace, which wasn=E2=80=99t used on the = bridge+members. This is from my own kernel config, but during testing with the GENERIC = kernel I had similar backtraces at reboot. I can=E2=80=99t do a lot right now, but I=E2=80=99m planning to: - build kernel with -O0 - do the deletem of the epair manually I=E2=80=99ll get back to you if I find something. Peter > On 23 Nov 2020, at 12:15, Kristof Provost wrote: >=20 > Peter, >=20 > Is that backtrace from the first or the second situation you describe? = What kernel config are you using with that backtrace? >=20 > This backtrace does not appear to involve the bridge. Given that part = of the panic message is cut off it=E2=80=99s very hard to conclude = anything at all from it. >=20 > Best regards, > Kristof >=20 > On 23 Nov 2020, at 11:52, Peter Blok wrote: >=20 >> Kristof, >>=20 >> With commit 367705+367706 and if_bridge statically linked. It crashes = while adding an epair of a jail. >>=20 >> With commit 367705+367706 and if_bridge dynamically loaded there is a = crash at reboot >>=20 >> #0 0xffffffff8069ddc5 at kdb_backtrace+0x65 >> #1 0xffffffff80652c8b at vpanic+0x17b >> #2 0xffffffff80652b03 at panic+0x43 >> #3 0xffffffff809c8951 at trap_fatal+0x391 >> #4 0xffffffff809c89af at trap_pfault+0x4f >> #5 0xffffffff809c7ff6 at trap+0x286 >> #6 0xffffffff809a1ec8 at calltrap+0x8 >> #7 0xffffffff8079f7ed at ip_input+0x63d >> #8 0xffffffff8077a07a at netisr_dispatch_src+0xca >> #9 0xffffffff8075a6f8 at ether_demux+0x138 >> #10 0xffffffff8075b9bb at ether_nh_input+0x33b >> #11 0xffffffff8077a07a at netisr_dispatch_src+0xca >> #12 0xffffffff8075ab1b at ether_input+0x4b >> #13 0xffffffff8077a80b at swi_net+0x12b >> #14 0xffffffff8061e10c at ithread_loop+0x23c >> #15 0xffffffff8061afbe at fork_exit+0x7e >> #16 0xffffffff809a2efe at fork_trampoline+0xe >>=20 >> Peter >>=20 >>> On 21 Nov 2020, at 17:22, Peter Blok wrote: >>>=20 >>> Kristof, >>>=20 >>> With a GENERIC kernel it does NOT happen. I do have a different = iflib related panic at reboot, but I=E2=80=99ll report that separately. >>>=20 >>> I brought the two config files closer together and found out that if = I remove if_bridge from the config file and have it loaded dynamically = when the bridge is created, the problem no longer happens and everything = works ok. >>>=20 >>> Peter >>>=20 >>>> On 20 Nov 2020, at 15:53, Kristof Provost wrote: >>>>=20 >>>> I still can=E2=80=99t reproduce that panic. >>>>=20 >>>> Does it happen immediately after you start a vnet jail? >>>>=20 >>>> Does it also happen with a GENERIC kernel? >>>>=20 >>>> Regards, >>>> Kristof >>>>=20 >>>> On 20 Nov 2020, at 14:53, Peter Blok wrote: >>>>=20 >>>>> The panic with ipsec code in the backtrace was already very = strange. I was using IPsec, but only on one interface totally separate = from the members of the bridge as well as the bridge itself. The jails = were not doing any ipsec as well. Note that panic was a while ago and it = was after the 1st bridge epochification was done on stable-12 which was = later backed out >>>>>=20 >>>>> Today the system is no longer using ipsec, but it is still = compiled in. I can remove it if need be for a test >>>>>=20 >>>>>=20 >>>>> src.conf >>>>> WITHOUT_KERBEROS=3Dyes >>>>> WITHOUT_GSSAPI=3Dyes >>>>> WITHOUT_SENDMAIL=3Dtrue >>>>> WITHOUT_MAILWRAPPER=3Dtrue >>>>> WITHOUT_DMAGENT=3Dtrue >>>>> WITHOUT_GAMES=3Dtrue >>>>> WITHOUT_IPFILTER=3Dtrue >>>>> WITHOUT_UNBOUND=3Dtrue >>>>> WITHOUT_PROFILE=3Dtrue >>>>> WITHOUT_ATM=3Dtrue >>>>> WITHOUT_BSNMP=3Dtrue >>>>> #WITHOUT_CROSS_COMPILER=3Dtrue >>>>> WITHOUT_DEBUG_FILES=3Dtrue >>>>> WITHOUT_DICT=3Dtrue >>>>> WITHOUT_FLOPPY=3Dtrue >>>>> WITHOUT_HTML=3Dtrue >>>>> WITHOUT_HYPERV=3Dtrue >>>>> WITHOUT_NDIS=3Dtrue >>>>> WITHOUT_NIS=3Dtrue >>>>> WITHOUT_PPP=3Dtrue >>>>> WITHOUT_TALK=3Dtrue >>>>> WITHOUT_TESTS=3Dtrue >>>>> WITHOUT_WIRELESS=3Dtrue >>>>> #WITHOUT_LIB32=3Dtrue >>>>> WITHOUT_LPR=3Dtrue >>>>>=20 >>>>> make.conf >>>>> KERNCONF=3DBHYVE >>>>> MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge = bridgestp if_vxlan pflog libmchain libiconv smbfs linux linux64 = linux_common linuxkpi linprocfs linsysfs ext2fs >>>>> DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 = python3=3D3.8 >>>>> OPTIONS_UNSET=3DDOCS NLS MANPAGES >>>>>=20 >>>>> BHYVE >>>>> cpu HAMMER >>>>> ident BHYVE >>>>>=20 >>>>> makeoptions DEBUG=3D-g # Build kernel with = gdb(1) debug symbols >>>>> makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for = DTrace support >>>>>=20 >>>>> options CAMDEBUG >>>>>=20 >>>>> options SCHED_ULE # ULE scheduler >>>>> options PREEMPTION # Enable kernel thread = preemption >>>>> options INET # InterNETworking >>>>> options INET6 # IPv6 communications protocols >>>>> options IPSEC >>>>> options TCP_OFFLOAD # TCP offload >>>>> options TCP_RFC7413 # TCP FASTOPEN >>>>> options SCTP # Stream Control Transmission = Protocol >>>>> options FFS # Berkeley Fast Filesystem >>>>> options SOFTUPDATES # Enable FFS soft updates = support >>>>> options UFS_ACL # Support for access control = lists >>>>> options UFS_DIRHASH # Improve performance on big = directories >>>>> options UFS_GJOURNAL # Enable gjournal-based UFS = journaling >>>>> options QUOTA # Enable disk quotas for UFS >>>>> options SUIDDIR >>>>> options NFSCL # Network Filesystem Client >>>>> options NFSD # Network Filesystem Server >>>>> options NFSLOCKD # Network Lock Manager >>>>> options MSDOSFS # MSDOS Filesystem >>>>> options CD9660 # ISO 9660 Filesystem >>>>> options FUSEFS >>>>> options NULLFS # NULL filesystem >>>>> options UNIONFS >>>>> options FDESCFS # File descriptor = filesystem >>>>> options PROCFS # Process filesystem (requires = PSEUDOFS) >>>>> options PSEUDOFS # Pseudo-filesystem framework >>>>> options GEOM_PART_GPT # GUID Partition Tables. >>>>> options GEOM_RAID # Soft RAID functionality. >>>>> options GEOM_LABEL # Provides labelization >>>>> options GEOM_ELI # Disk encryption. >>>>> options COMPAT_FREEBSD32 # Compatible with i386 binaries >>>>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >>>>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >>>>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >>>>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >>>>> options COMPAT_FREEBSD9 # Compatible with FreeBSD9 >>>>> options COMPAT_FREEBSD10 # Compatible with FreeBSD10 >>>>> options COMPAT_FREEBSD11 # Compatible with FreeBSD11 >>>>> options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI >>>>> options KTRACE # ktrace(1) support >>>>> options STACK # stack(9) support >>>>> options SYSVSHM # SYSV-style shared memory >>>>> options SYSVMSG # SYSV-style message queues >>>>> options SYSVSEM # SYSV-style semaphores >>>>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions >>>>> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. >>>>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >>>>> options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) >>>>> options AUDIT # Security event auditing >>>>> options CAPABILITY_MODE # Capsicum capability mode >>>>> options CAPABILITIES # Capsicum capabilities >>>>> options MAC # TrustedBSD MAC Framework >>>>> options MAC_PORTACL >>>>> options MAC_NTPD >>>>> options KDTRACE_FRAME # Ensure frames are compiled in >>>>> options KDTRACE_HOOKS # Kernel DTrace hooks >>>>> options DDB_CTF # Kernel ELF linker loads CTF = data >>>>> options INCLUDE_CONFIG_FILE # Include this file in kernel >>>>>=20 >>>>> # Debugging support. Always need this: >>>>> options KDB # Enable kernel debugger = support. >>>>> options KDB_TRACE # Print a stack trace for a = panic. >>>>> options KDB_UNATTENDED >>>>>=20 >>>>> # Make an SMP-capable kernel by default >>>>> options SMP # Symmetric MultiProcessor = Kernel >>>>> options EARLY_AP_STARTUP >>>>>=20 >>>>> # CPU frequency control >>>>> device cpufreq >>>>> device cpuctl >>>>> device coretemp >>>>>=20 >>>>> # Bus support. >>>>> device acpi >>>>> options ACPI_DMAR >>>>> device pci >>>>> options PCI_IOV # PCI SR-IOV support >>>>>=20 >>>>> device iicbus >>>>> device iicbb >>>>>=20 >>>>> device iic >>>>> device ic >>>>> device iicsmb >>>>>=20 >>>>> device ichsmb >>>>> device smbus >>>>> device smb >>>>>=20 >>>>> #device jedec_dimm >>>>>=20 >>>>> # ATA controllers >>>>> device ahci # AHCI-compatible SATA = controllers >>>>> device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA >>>>>=20 >>>>> # SCSI Controllers >>>>> device mps # LSI-Logic MPT-Fusion 2 >>>>>=20 >>>>> # ATA/SCSI peripherals >>>>> device scbus # SCSI bus (required for = ATA/SCSI) >>>>> device da # Direct Access (disks) >>>>> device cd # CD >>>>> device pass # Passthrough device = (direct ATA/SCSI access) >>>>> device ses # Enclosure Services = (SES and SAF-TE) >>>>> device sg >>>>>=20 >>>>> device cfiscsi >>>>> device ctl # CAM Target Layer >>>>> device iscsi >>>>>=20 >>>>> # atkbdc0 controls both the keyboard and the PS/2 mouse >>>>> device atkbdc # AT keyboard controller >>>>> device atkbd # AT keyboard >>>>> device psm # PS/2 mouse >>>>>=20 >>>>> device kbdmux # keyboard multiplexer >>>>>=20 >>>>> # vt is the new video console driver >>>>> device vt >>>>> device vt_vga >>>>> device vt_efifb >>>>>=20 >>>>> # Serial (COM) ports >>>>> device uart # Generic UART driver >>>>>=20 >>>>> # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure >>>>> device iflib >>>>> device em # Intel PRO/1000 Gigabit = Ethernet Family >>>>> device ix # Intel PRO/10GbE PCIE = PF Ethernet >>>>>=20 >>>>> # Network stack virtualization. >>>>> options VIMAGE >>>>>=20 >>>>> # Pseudo devices. >>>>> device crypto >>>>> device cryptodev >>>>> device loop # Network loopback >>>>> device random # Entropy device >>>>> device padlock_rng # VIA Padlock RNG >>>>> device rdrand_rng # Intel Bull Mountain = RNG >>>>> device ipmi >>>>> device smbios >>>>> device vpd >>>>> device aesni # AES-NI OpenCrypto = module >>>>> device ether # Ethernet support >>>>> device lagg >>>>> device vlan # 802.1Q VLAN support >>>>> device tuntap # Packet tunnel. >>>>> device md # Memory "disks" >>>>> device gif # IPv6 and IPv4 = tunneling >>>>> device firmware # firmware assist module >>>>>=20 >>>>> device pf >>>>> #device pflog >>>>> #device pfsync >>>>>=20 >>>>> # The `bpf' device enables the Berkeley Packet Filter. >>>>> # Be aware of the administrative consequences of enabling this! >>>>> # Note that 'bpf' is required for DHCP. >>>>> device bpf # Berkeley packet filter >>>>>=20 >>>>> # The `epair' device implements a virtual back-to-back connected = Ethernet >>>>> # like interface pair. >>>>> device epair >>>>>=20 >>>>> # USB support >>>>> options USB_DEBUG # enable debug msgs >>>>> device uhci # UHCI PCI->USB = interface >>>>> device ohci # OHCI PCI->USB = interface >>>>> device ehci # EHCI PCI->USB = interface (USB 2.0) >>>>> device xhci # XHCI PCI->USB = interface (USB 3.0) >>>>> device usb # USB Bus (required) >>>>> device uhid >>>>> device ukbd # Keyboard >>>>> device umass # Disks/Mass storage - = Requires scbus and da >>>>> device ums >>>>>=20 >>>>> device filemon >>>>>=20 >>>>> device if_bridge >>>>>=20 >>>>>> On 20 Nov 2020, at 12:53, Kristof Provost wrote: >>>>>>=20 >>>>>> Can you share your kernel config file (and src.conf / make.conf = if they exist)? >>>>>>=20 >>>>>> This second panic is in the IPSec code. My current thinking is = that your kernel config is triggering a bug that=E2=80=99s manifesting = in multiple places, but not actually caused by those places. >>>>>>=20 >>>>>> I=E2=80=99d like to be able to reproduce it so we can debug it. >>>>>>=20 >>>>>> Best regards, >>>>>> Kristof >>>>>>=20 >>>>>> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>>>>>> Hi Kristof, >>>>>>>=20 >>>>>>> This is 12-stable. With the previous bridge epochification that = was backed out my config had a panic too. >>>>>>>=20 >>>>>>> I don=E2=80=99t have any local modifications. I did a clean = rebuild after removing /usr/obj/usr >>>>>>>=20 >>>>>>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko = and nmdm.ko as modules. Everything else is statically linked. I have = removed all drivers not needed for the hardware at hand. >>>>>>>=20 >>>>>>> My bridge is between two vlans from the same trunk and the jail = epair devices as well as the bhyve tap devices. >>>>>>>=20 >>>>>>> The panic happens when the jails are starting. >>>>>>>=20 >>>>>>> I can try to narrow it down over the weekend and make the crash = dump available for analysis. >>>>>>>=20 >>>>>>> Previously I had the following crash with 363492 >>>>>>>=20 >>>>>>> kernel trap 12 with interrupts disabled >>>>>>>=20 >>>>>>>=20 >>>>>>> Fatal trap 12: page fault while in kernel mode >>>>>>> cpuid =3D 2; apic id =3D 02 >>>>>>> fault virtual address =3D 0xffffffff00000410 >>>>>>> fault code =3D supervisor read data, page not = present >>>>>>> instruction pointer =3D 0x20:0xffffffff80692326 >>>>>>> stack pointer =3D 0x28:0xfffffe00c06097b0 >>>>>>> frame pointer =3D 0x28:0xfffffe00c06097f0 >>>>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>>>> =3D DPL 0, pres 1, long 1, def32 0, gran = 1 >>>>>>> processor eflags =3D resume, IOPL =3D 0 >>>>>>> current process =3D 2030 (ifconfig) >>>>>>> trap number =3D 12 >>>>>>> panic: page fault >>>>>>> cpuid =3D 2 >>>>>>> time =3D 1595683412 >>>>>>> KDB: stack backtrace: >>>>>>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>>>>>> #1 0xffffffff8064d67b at vpanic+0x17b >>>>>>> #2 0xffffffff8064d4f3 at panic+0x43 >>>>>>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>>>>>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>>>>>> #5 0xffffffff809cb9b6 at trap+0x286 >>>>>>> #6 0xffffffff809a5b28 at calltrap+0x8 >>>>>>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>>>>>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>>>>>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>>>>>> #10 0xffffffff8075274f at ifioctl+0x47f >>>>>>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>>>>>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>>>>>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>>>>>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>> On 20 Nov 2020, at 11:30, Kristof Provost = wrote: >>>>>>>>=20 >>>>>>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = wrote: >>>>>>>>> I=E2=80=99m afraid the last Epoch fix for bridge is not = solving the problem ( or perhaps creates a new ). >>>>>>>>>=20 >>>>>>>> We=E2=80=99re talking about the stable/12 branch, right? >>>>>>>>=20 >>>>>>>>> This seems to happen when the jail epair is added to the = bridge. >>>>>>>>>=20 >>>>>>>> There must be something more to it than that. I=E2=80=99ve run = the bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>>>>>>=20 >>>>>>>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>>>>>>=20 >>>>>>>>> kernel trap 12 with interrupts disabled >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Fatal trap 12: page fault while in kernel mode >>>>>>>>> cpuid =3D 6; apic id =3D 06 >>>>>>>>> fault virtual address =3D 0xc10 >>>>>>>>> fault code =3D supervisor read data, page not = present >>>>>>>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>>>>>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>>>>>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>>>>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>>>>>> =3D DPL 0, pres 1, long 1, def32 0, gran = 1 >>>>>>>>> processor eflags =3D resume, IOPL =3D 0 >>>>>>>>> current process =3D 1686 (jail) >>>>>>>>> trap number =3D 12 >>>>>>>>> panic: page fault >>>>>>>>> cpuid =3D 6 >>>>>>>>> time =3D 1605811310 >>>>>>>>> KDB: stack backtrace: >>>>>>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>>>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>>>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>>>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>>>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>>>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>>>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>>>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>>>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>>>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>>>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>>>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>>>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>>>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>>>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>>>>>>=20 >>>>>>>> This panic is rather odd. This isn=E2=80=99t even the bridge = code. This is during initial creation of the vnet. I don=E2=80=99t = really see how this could even trigger panics. >>>>>>>> That panic looks as if something corrupted the = net_epoch_preempt, by overwriting the epoch->e_epoch. The bridge patches = only access this variable through the well-established functions and = macros. I see no obvious way that they could corrupt it. >>>>>>>>=20 >>>>>>>> Best regards, >>>>>>>> Kristof >>>>>>=20 >>>>>>=20 >>>>>> _______________________________________________ >>>>>> freebsd-stable@freebsd.org mailing list >>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>>=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" --Apple-Mail=_2AEED608-5094-46C3-AD01-832A8DF68E09 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIzMTIyNzE5WjAvBgkqhkiG9w0BCQQxIgQg3thHYarc P71Ax5cclA5TEoXgfsjzFI73++gt+jLXcKEwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAGe0uOPjpiCzg+anYfq1F5OJDlvioMjm xYq4/juolJXpJYWoPTssFiDCSgnm5LtK0lS4LZ6ZT6l5w0L80dGKre911X9SxxjyfyG1vC4j2o0e jA7gWHvJMeg6atDy1RV2tYZ48Fkz/mVRWdfBAyovMIjOTEPVsyqMkq8+rn2Q4BsO5ipCucf/l1OD Letb0was5xH4MHRdvmeoLh6Q3BgKZrad4fvAve2xfe7pKmJqUxKFP6f2rc6P740Hm9SqamsPWloW kpamccPKapb/nlM+sw9X1FxeDdJBtsWKk8+RCbI6CynePIIjHOD80KP8xgI8NUiTAWV9r52W03PA T4hL4DUAAAAAAAA= --Apple-Mail=_2AEED608-5094-46C3-AD01-832A8DF68E09-- From owner-freebsd-stable@freebsd.org Fri Nov 27 15:03:17 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 71BC64A9652 for ; Fri, 27 Nov 2020 15:03:17 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CjHts2tlnz3PM5 for ; Fri, 27 Nov 2020 15:03:17 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 2C29D2D665 for ; Fri, 27 Nov 2020 15:03:17 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.230] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 778343978 for ; Fri, 27 Nov 2020 18:03:14 +0300 (MSK) To: FreeBSD Stable Reply-To: lev@FreeBSD.org From: Lev Serebryakov Subject: 12-STABLE try to init thead-using libraries before threads and program crashes Organization: FreeBSD Message-ID: Date: Fri, 27 Nov 2020 18:03:13 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: base64 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2020 15:03:17 -0000 DQogIEkgaGF2ZSBsb2NhbGx5LWJ1aWx0IG5ldC9zYW1iYTQxMyBwb3J0IG9uIDEyLVNUQUJM RSAocjM2NzkzNykgd2hpY2ggY3Jhc2hlcyBpbiBsaWJyYXJ5IGluaXRpYWxpemF0aW9uIGNv ZGUgZHVlIHRvIHdyb25nIGxpYnJhcnkgaW5pdGlhbGl6YXRpb24gb3JkZXI6DQoNCihObyBk ZWJ1Z2dpbmcgc3ltYm9scyBmb3VuZCBpbiAvdXNyL2xvY2FsL2Jpbi90ZXN0cGFybSkNCihn ZGIpIGIgIF9saWJwdGhyZWFkX2luaXQNCkZ1bmN0aW9uICJfbGlicHRocmVhZF9pbml0IiBu b3QgZGVmaW5lZC4NCk1ha2UgYnJlYWtwb2ludCBwZW5kaW5nIG9uIGZ1dHVyZSBzaGFyZWQg bGlicmFyeSBsb2FkPyAoeSBvciBbbl0pIHkNCkJyZWFrcG9pbnQgMSAoX2xpYnB0aHJlYWRf aW5pdCkgcGVuZGluZy4NCihnZGIpIHJ1bg0KU3RhcnRpbmcgcHJvZ3JhbTogL3Vzci9sb2Nh bC9iaW4vdGVzdHBhcm0NCg0KUHJvZ3JhbSByZWNlaXZlZCBzaWduYWwgU0lHU0VHViwgU2Vn bWVudGF0aW9uIGZhdWx0Lg0KdGhyX21hbGxvY19sb2NrIChjdXJ0aHJlYWQ9MHg4MDFlMDc3 ZDApIGF0IC91c3Ivc3JjL2xpYi9saWJ0aHIvdGhyZWFkL3Rocl9tYWxsb2MuYzo2Ng0KNjYg ICAgICAgICAgICAgIGN1cnRocmVhZC0+bG9ja2xldmVsKys7DQooZ2RiKSBidA0KIzAgIHRo cl9tYWxsb2NfbG9jayAoY3VydGhyZWFkPTB4ODAxZTA3N2QwKSBhdCAvdXNyL3NyYy9saWIv bGlidGhyL3RocmVhZC90aHJfbWFsbG9jLmM6NjYNCiMxICBfX3Rocl9jYWxsb2MgKG51bT0x LCBzaXplPTk2KSBhdCAvdXNyL3NyYy9saWIvbGlidGhyL3RocmVhZC90aHJfbWFsbG9jLmM6 ODgNCiMyICAweDAwMDAwMDA4MDE0NzQ4NDMgaW4gbXV0ZXhfaW5pdCAobXV0ZXg9MHg4MDEw NzIwMDgsIG11dGV4X2F0dHI9PG9wdGltaXplZCBvdXQ+LCBjYWxsb2NfY2I9PG9wdGltaXpl ZCBvdXQ+KSBhdCAvdXNyL3NyYy9saWIvbGlidGhyL3RocmVhZC90aHJfbXV0ZXguYzoyOTUN CiMzICBfX1R0aHJfbXV0ZXhfaW5pdCAobXV0ZXg9MHg4MDEwNzIwMDgsIG11dGV4X2F0dHI9 PG9wdGltaXplZCBvdXQ+KSBhdCAvdXNyL3NyYy9saWIvbGlidGhyL3RocmVhZC90aHJfbXV0 ZXguYzozOTUNCiM0ICAweDAwMDAwMDA4MDE2ZDYyZmMgaW4gPz8gKCkgZnJvbSAvdXNyL2xv Y2FsL2xpYi9saWJnbnV0bHMuc28uMzANCiM1ICAweDAwMDAwMDA4MDE2Y2ZjYjMgaW4gPz8g KCkgZnJvbSAvdXNyL2xvY2FsL2xpYi9saWJnbnV0bHMuc28uMzANCiM2ICAweDAwMDAwMDA4 MDE2ZDAwNzcgaW4gPz8gKCkgZnJvbSAvdXNyL2xvY2FsL2xpYi9saWJnbnV0bHMuc28uMzAN CiM3ICAweDAwMDAwMDA4MDEwMzczMGQgaW4gb2JqbGlzdF9jYWxsX2luaXQgKGxpc3Q9PG9w dGltaXplZCBvdXQ+LCBsb2Nrc3RhdGU9PG9wdGltaXplZCBvdXQ+KSBhdCAvdXNyL3NyYy9s aWJleGVjL3J0bGQtZWxmL3J0bGQuYzoyODIzDQojOCAgMHgwMDAwMDAwODAxMDM2MDNkIGlu IF9ydGxkIChzcD0weDdmZmZmZmZmZWI1OCwgZXhpdF9wcm9jPTB4N2ZmZmZmZmZlYjIwLCBv YmpwPTB4N2ZmZmZmZmZlYjI4KSBhdCAvdXNyL3NyYy9saWJleGVjL3J0bGQtZWxmL3J0bGQu Yzo4MTENCiM5ICAweDAwMDAwMDA4MDEwMzM4YzkgaW4gcnRsZF9zdGFydCAoKSBhdCAvdXNy L3NyYy9saWJleGVjL3J0bGQtZWxmL2FtZDY0L3J0bGRfc3RhcnQuUzozOQ0KIzEwIDB4MDAw MDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KKGdkYikNCg0KICBQbGVhc2Ugbm90ZSwgdGhhdCBg X2xpYnB0aHJlYWRfaW5pdGAgSEFTIEJFRU4gTk9UIENBTExFRCBiZWZvcmUgYF9UdGhyX211 dGV4X2luaXRgLg0KDQogIExvb2tzIGxpa2Ugc29tZSBjb3JuZXItY2FzZSBwcm9ibGVtIGlu IHJ0bGQ/DQoNCiAgTGluayBjb21tYW5kIGZvciB0aGlzIHByb2dyYW0gaXM6DQoNClszNTE3 LzM2NjBdIExpbmtpbmcgYmluL2RlZmF1bHQvc291cmNlMy91dGlscy90ZXN0cGFybQ0KcnVu bmVyIFsnY2MnLCAnc291cmNlMy91dGlscy90ZXN0cGFybS5jLjQxLm8nLCAnLW8vd3JrZGly cy91c3IvcG9ydHMvbmV0L3NhbWJhNDEzL3dvcmsvc2FtYmEtNC4xMy4xL2Jpbi9kZWZhdWx0 L3NvdXJjZTMvdXRpbHMvdGVzdHBhcm0nLCAnLVdsLC1Cc3RhdGljJywgJy1XbCwtQmR5bmFt aWMnLCAnLUwvd3JrZGlycy91c3IvcG9ydHMvbmV0L3NhbWJhNDEzL3dvcmsvc2FtYmEtNC4x My4xL2Jpbi9kZWZhdWx0L3NvdXJjZTQvaGVpbWRhbF9idWlsZCcsICctTC93cmtkaXJzL3Vz ci9wb3J0cy9uZXQvc2FtYmE0MTMvd29yay9zYW1iYS00LjEzLjEvYmluL2RlZmF1bHQvc291 cmNlNC9saWIvZXZlbnRzJywgJy1ML3dya2RpcnMvdXNyL3BvcnRzL25ldC9zYW1iYTQxMy93 b3JrL3NhbWJhLTQuMTMuMS9iaW4vZGVmYXVsdC9saWIvdGRiX3dyYXAnLCAnLUwvd3JrZGly cy91c3IvcG9ydHMvbmV0L3NhbWJhNDEzL3dvcmsvc2FtYmEtNC4xMy4xL2Jpbi9kZWZhdWx0 L2xpYmNsaS9zZWN1cml0eScsICctTC93cmtkaXJzL3Vzci9wb3J0cy9uZXQvc2FtYmE0MTMv d29yay9zYW1iYS00LjEzLjEvYmluL2RlZmF1bHQvbGlicnBjJywgJy1ML3dya2RpcnMvdXNy L3BvcnRzL25ldC9zYW1iYTQxMy93b3JrL3NhbWJhLTQuMTMuMS9iaW4vZGVmYXVsdC9saWJj bGkvcmVnaXN0cnknLCAnLUwvd3JrZGlycy91c3IvcG9ydHMvbmV0L3NhbWJhNDEzL3dvcmsv c2FtYmEtNC4xMy4xL2Jpbi9kZWZhdWx0L2xpYicsICctTC93cmtkaXJzL3Vzci9wb3J0cy9u ZXQvc2FtYmE0MTMvd29yay9zYW1iYS00LjEzLjEvYmluL2RlZmF1bHQvbGliL2Rid3JhcCcs ICctTC93cmtkaXJzL3Vzci9wb3J0cy9uZXQvc2FtYmE0MTMvd29yay9zYW1iYS00LjEzLjEv YmluL2RlZmF1bHQvbGliL3NvY2tldCcsICctTC93cmtkaXJzL3Vzci9wb3J0cy9uZXQvc2Ft YmE0MTMvd29yay9zYW1iYS00LjEzLjEvYmluL2RlZmF1bHQvbGliL3BhcmFtJywgJy1ML3dy a2RpcnMvdXNyL3BvcnRzL25ldC9zYW1iYTQxMy93b3JrL3NhbWJhLTQuMTMuMS9iaW4vZGVm YXVsdC9saWIvbWVzc2FnaW5nJywgJy1ML3dya2RpcnMvdXNyL3BvcnRzL25ldC9zYW1iYTQx My93b3JrL3NhbWJhLTQuMTMuMS9iaW4vZGVmYXVsdC9saWIvdXRpbCcsICctTC93cmtkaXJz L3Vzci9wb3J0cy9uZXQvc2FtYmE0MTMvd29yay9zYW1iYS00LjEzLjEvYmluL2RlZmF1bHQv bGliY2xpL3V0aWwnLCAnLUwvd3JrZGlycy91c3IvcG9ydHMvbmV0L3NhbWJhNDEzL3dvcmsv c2FtYmEtNC4xMy4xL2Jpbi9kZWZhdWx0L2xpYi9yZXBsYWNlJywgJy1ML3dya2RpcnMvdXNy L3BvcnRzL25ldC9zYW1iYTQxMy93b3JrL3NhbWJhLTQuMTMuMS9iaW4vZGVmYXVsdC9zb3Vy Y2UzJywgJy1ML3Vzci9sb2NhbC9saWInLCAnLUwvdXNyL2xvY2FsL2xpYicsICctTC91c3Iv bG9jYWwvbGliJywgJy1ML3Vzci9sb2NhbC9saWInLCAnLUwvdXNyL2xvY2FsL2xpYicsICct TC91c3IvbG9jYWwvbGliJywgJy1scG9wdC1zYW1iYTMtc2FtYmE0JywgJy1sc21iY29uZics ICctbHJlcGxhY2Utc2FtYmE0JywgJy1sc2FtYmEtZXJyb3JzJywgJy1sY21kbGluZS1jb250 ZXh0cy1zYW1iYTQnLCAnLWxzYW1iYS11dGlsJywgJy1sc2FtYmEzLXV0aWwtc2FtYmE0Jywg Jy1sbWVzc2FnZXMtZGdtLXNhbWJhNCcsICctbHN5cy1ydy1zYW1iYTQnLCAnLWxtZXNzYWdl cy11dGlsLXNhbWJhNCcsICctbGlvdi1idWYtc2FtYmE0JywgJy1sc2FtYmEtaG9zdGNvbmZp ZycsICctbHNvY2tldC1ibG9ja2luZy1zYW1iYTQnLCAnLWxpbnRlcmZhY2VzLXNhbWJhNCcs ICctbGRid3JhcC1zYW1iYTQnLCAnLWx0ZXZlbnQtdXRpbCcsICctbHNhbWJhLXNvY2tldHMt c2FtYmE0JywgJy1sdXRpbC1yZWctc2FtYmE0JywgJy1sdXRpbC10ZGItc2FtYmE0JywgJy1s bmRyJywgJy1sdGFsbG9jLXJlcG9ydC1wcmludGYtc2FtYmE0JywgJy1sc2VydmVyLWlkLWRi LXNhbWJhNCcsICctbHNhbWJhLWNsdXN0ZXItc3VwcG9ydC1zYW1iYTQnLCAnLWxDSEFSU0VU My1zYW1iYTQnLCAnLWxzYW1iYS1zZWN1cml0eS1zYW1iYTQnLCAnLWxzbWJkLXNoaW0tc2Ft YmE0JywgJy1sc2FtYmEtZGVidWctc2FtYmE0JywgJy1sZ2VucmFuZC1zYW1iYTQnLCAnLWx0 aW1lLWJhc2ljLXNhbWJhNCcsICctbHV0aWwtc2V0aWQtc2FtYmE0JywgJy1sbXNnaGRyLXNh bWJhNCcsICctbHNlcnZlci1yb2xlLXNhbWJhNCcsICctbHRkYi13cmFwLXNhbWJhNCcsICct bGV2ZW50cy1zYW1iYTQnLCAnLWxuZHItbmJ0JywgJy1scm9rZW4tc2FtYmE0JywgJy1sZXhl Y2luZm8nLCAnLWx0ZXZlbnQnLCAnLWx0YWxsb2MnLCAnLWxwdGhyZWFkJywgJy1sdXRpbCcs ICctbHVud2luZC1nZW5lcmljJywgJy1sdW53aW5kJywgJy1saWNvbnYnLCAnLWx6JywgJy1s dGRiJywgJy1scG9wdCcsICctbGdudXRscycsICctbHRhbGxvYycsICctZnN0YWNrLXByb3Rl Y3Rvci1zdHJvbmcnLCAnLUwvdXNyL2xvY2FsL2xpYicsICctcGllJywgJy1XbCwteixyZWxy bywteixub3cnLCAnLVdsLC1uby11bmRlZmluZWQnLCAnLVdsLC0tZXhwb3J0LWR5bmFtaWMn XQ0KDQotLSANCi8vIExldiBTZXJlYnJ5YWtvdg0K From owner-freebsd-stable@freebsd.org Fri Nov 27 17:03:53 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 215A44AC6D2 for ; Fri, 27 Nov 2020 17:03:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CjLZ057Dsz3lZn; Fri, 27 Nov 2020 17:03:52 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 0ARH3a35079232 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 27 Nov 2020 19:03:39 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 0ARH3a35079232 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 0ARH3aEa079231; Fri, 27 Nov 2020 19:03:36 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 27 Nov 2020 19:03:36 +0200 From: Konstantin Belousov To: Lev Serebryakov Cc: FreeBSD Stable Subject: Re: 12-STABLE try to init thead-using libraries before threads and program crashes Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4CjLZ057Dsz3lZn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2020 17:03:53 -0000 On Fri, Nov 27, 2020 at 06:03:13PM +0300, Lev Serebryakov wrote: >=20 > I have locally-built net/samba413 port on 12-STABLE (r367937) which cras= hes in library initialization code due to wrong library initialization orde= r: >=20 > (No debugging symbols found in /usr/local/bin/testparm) > (gdb) b _libpthread_init > Function "_libpthread_init" not defined. > Make breakpoint pending on future shared library load? (y or [n]) y > Breakpoint 1 (_libpthread_init) pending. > (gdb) run > Starting program: /usr/local/bin/testparm >=20 > Program received signal SIGSEGV, Segmentation fault. > thr_malloc_lock (curthread=3D0x801e077d0) at /usr/src/lib/libthr/thread/t= hr_malloc.c:66 > 66 curthread->locklevel++; > (gdb) bt > #0 thr_malloc_lock (curthread=3D0x801e077d0) at /usr/src/lib/libthr/thre= ad/thr_malloc.c:66 > #1 __thr_calloc (num=3D1, size=3D96) at /usr/src/lib/libthr/thread/thr_m= alloc.c:88 > #2 0x0000000801474843 in mutex_init (mutex=3D0x801072008, mutex_attr=3D<= optimized out>, calloc_cb=3D) at /usr/src/lib/libthr/thread/= thr_mutex.c:295 > #3 __Tthr_mutex_init (mutex=3D0x801072008, mutex_attr=3D)= at /usr/src/lib/libthr/thread/thr_mutex.c:395 > #4 0x00000008016d62fc in ?? () from /usr/local/lib/libgnutls.so.30 > #5 0x00000008016cfcb3 in ?? () from /usr/local/lib/libgnutls.so.30 > #6 0x00000008016d0077 in ?? () from /usr/local/lib/libgnutls.so.30 > #7 0x000000080103730d in objlist_call_init (list=3D, lock= state=3D) at /usr/src/libexec/rtld-elf/rtld.c:2823 > #8 0x000000080103603d in _rtld (sp=3D0x7fffffffeb58, exit_proc=3D0x7ffff= fffeb20, objp=3D0x7fffffffeb28) at /usr/src/libexec/rtld-elf/rtld.c:811 > #9 0x00000008010338c9 in rtld_start () at /usr/src/libexec/rtld-elf/amd6= 4/rtld_start.S:39 > #10 0x0000000000000000 in ?? () > (gdb) >=20 > Please note, that `_libpthread_init` HAS BEEN NOT CALLED before `_Tthr_m= utex_init`. >=20 > Looks like some corner-case problem in rtld? >=20 > Link command for this program is: >=20 > [3517/3660] Linking bin/default/source3/utils/testparm > runner ['cc', 'source3/utils/testparm.c.41.o', '-o/wrkdirs/usr/ports/net/= samba413/work/samba-4.13.1/bin/default/source3/utils/testparm', '-Wl,-Bstat= ic', '-Wl,-Bdynamic', '-L/wrkdirs/usr/ports/net/samba413/work/samba-4.13.1/= bin/default/source4/heimdal_build', '-L/wrkdirs/usr/ports/net/samba413/work= /samba-4.13.1/bin/default/source4/lib/events', '-L/wrkdirs/usr/ports/net/sa= mba413/work/samba-4.13.1/bin/default/lib/tdb_wrap', '-L/wrkdirs/usr/ports/n= et/samba413/work/samba-4.13.1/bin/default/libcli/security', '-L/wrkdirs/usr= /ports/net/samba413/work/samba-4.13.1/bin/default/librpc', '-L/wrkdirs/usr/= ports/net/samba413/work/samba-4.13.1/bin/default/libcli/registry', '-L/wrkd= irs/usr/ports/net/samba413/work/samba-4.13.1/bin/default/lib', '-L/wrkdirs/= usr/ports/net/samba413/work/samba-4.13.1/bin/default/lib/dbwrap', '-L/wrkdi= rs/usr/ports/net/samba413/work/samba-4.13.1/bin/default/lib/socket', '-L/wr= kdirs/usr/ports/net/samba413/work/samba-4.13.1/bin/default/lib/param', '-L/= wrkdirs/usr/ports/net/sam > ba413/work/samba-4.13.1/bin/default/lib/messaging', '-L/wrkdirs/usr/ports= /net/samba413/work/samba-4.13.1/bin/default/lib/util', '-L/wrkdirs/usr/port= s/net/samba413/work/samba-4.13.1/bin/default/libcli/util', '-L/wrkdirs/usr/= ports/net/samba413/work/samba-4.13.1/bin/default/lib/replace', '-L/wrkdirs/= usr/ports/net/samba413/work/samba-4.13.1/bin/default/source3', '-L/usr/loca= l/lib', '-L/usr/local/lib', '-L/usr/local/lib', '-L/usr/local/lib', '-L/usr= /local/lib', '-L/usr/local/lib', '-lpopt-samba3-samba4', '-lsmbconf', '-lre= place-samba4', '-lsamba-errors', '-lcmdline-contexts-samba4', '-lsamba-util= ', '-lsamba3-util-samba4', '-lmessages-dgm-samba4', '-lsys-rw-samba4', '-lm= essages-util-samba4', '-liov-buf-samba4', '-lsamba-hostconfig', '-lsocket-b= locking-samba4', '-linterfaces-samba4', '-ldbwrap-samba4', '-ltevent-util',= '-lsamba-sockets-samba4', '-lutil-reg-samba4', '-lutil-tdb-samba4', '-lndr= ', '-ltalloc-report-printf-samba4', '-lserver-id-db-samba4', '-lsamba-clust= er-support-samba4', '-lC > HARSET3-samba4', '-lsamba-security-samba4', '-lsmbd-s > him-samba4', '-lsamba-debug-samba4', '-lgenrand-samba4', '-ltime-basic-sa= mba4', '-lutil-setid-samba4', '-lmsghdr-samba4', '-lserver-role-samba4', '-= ltdb-wrap-samba4', '-levents-samba4', '-lndr-nbt', '-lroken-samba4', '-lexe= cinfo', '-ltevent', '-ltalloc', '-lpthread', '-lutil', '-lunwind-generic', = '-lunwind', '-liconv', '-lz', '-ltdb', '-lpopt', '-lgnutls', '-ltalloc', '-= fstack-protector-strong', '-L/usr/local/lib', '-pie', '-Wl,-z,relro,-z,now'= , '-Wl,-no-undefined', '-Wl,--export-dynamic'] >=20 libthr is cleanly linked too early, it should come after all consumers. Anyway, try this. diff --git a/lib/libthr/thread/thr_mutex.c b/lib/libthr/thread/thr_mutex.c index 57984ef6d0e..303386db7fe 100644 --- a/lib/libthr/thread/thr_mutex.c +++ b/lib/libthr/thread/thr_mutex.c @@ -384,6 +384,8 @@ __Tthr_mutex_init(pthread_mutex_t * __restrict mutex, struct pthread_mutex *pmtx; int ret; =20 + _thr_check_init(); + if (mutex_attr !=3D NULL) { ret =3D mutex_check_attr(*mutex_attr); if (ret !=3D 0)