From owner-freebsd-net@freebsd.org Wed May 5 16:35:35 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7174B5FC0DF for ; Wed, 5 May 2021 16:35:35 +0000 (UTC) (envelope-from schmiedgen@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fb2Py2WS4z4WCq; Wed, 5 May 2021 16:35:33 +0000 (UTC) (envelope-from schmiedgen@gmx.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1620232532; bh=lDrfy6GglOQsHP486JeueRwdQufgI3QSpmqw6XaaTsc=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=Vwf845fN2YzFjD/3wEh800qMK+9TDhQgESnIgAahDDlmxl//pgJV0cJdBGpWNWuxE 8un7Ghpcpl+9VlvdLT44/Bp47CdqiQevS8MZbs4y8098lxoJjsA9KA0v4MgMq5Q4/O gK6/+gkzpdXOuiOI30gaHtXuk11AsQs4AZke7YSA= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.10.5] ([62.246.110.10]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N4QsO-1lVunS0dok-011SFJ; Wed, 05 May 2021 18:35:32 +0200 Subject: Re: page fault while in kernel mode - after upgrade from 12.2 to 13.0 To: Mark Johnston Cc: freebsd-net@freebsd.org References: From: Michael Schmiedgen Message-ID: <51a3abc5-76b9-df09-acbe-895b62ec87b3@gmx.net> Date: Wed, 5 May 2021 18:35:32 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:osiUl7OJ2fm8PzqnM2nhlvlplpHJwipSEl+JH1Ov5om0vHV0Zt4 F5DDb7tLEyi7hE50Iz4NYQyWosWp/nIFrUg6+TVYUPPkcpJmkT0VIt3ajQ1/gvi3iMhaxCx WLVdoZII3xg1FYTOS/KF21Z4HDfPTMiuAVUYmTcwrGOlygKSN/qcSkZX6F6eMLDXN7yckDU dBJcfDvFJKVe38c47XsTw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:VxMxmDpONcA=:u8IQMYkfrynCsg3/NZwd2Z M6kAnuMX1rEbwUtq8fv9OCfpE8nTo1tGSig3z0yAWzINFzsOp33S8/nxnfhxPZ6PV7PKIZz7g jM7oZ1WcfOJ6vQtDVZvD/x+wLD7AUl7m8MBytSRXdkCwCSgRXfTM08k0tg0R2Xwz87b2NpX7j UFP4UY8iBKkNf7s7/sdUXBarcLXABpvlM0bNrZ6lAk0ucEPHw39t9WW54uybsn90dAUbgDqk2 ocVBcbi0NqHMZkKvC60mQBMtsXXcLnce8XFuvqZd3qaX5OyoQztEJqAR1AZGhwN8qToJpWb/K CnQtCm4l7/aiz6B/6xiUwLquORevZqQAAn3F/Z3xsg73ZanYTEXYMSzlz2JC7H+2CxysXCfVh Zf/L7WCQXPudzc6G46hTAv/j2YDoqio3NAqVF3/HziSdLZip1WYg7ehKmDU+m1CtrHB9JtSeG Q4mcfGUkeCZsPEKcBXi6j4/nqC7meth3kN2hw1A9rcsIvS6+z1yYH90EuYeTTaIpE9wsG6a2H Z/XHckzptQ8uWY4yCoriyStR9MMDl8/xYgB5ga/6zf1YbtSp89x+pW3eHV1IqR/BvvkIquKgn c3PT19zM6YmzrJELyFBZz3Uxb1d9nDbGkKnwGKq/NAopuK1b0iJwiFnYLXfjLeE+o1jorY35P Iz7dPFkC+eBsbxu+fHVchNG1suwfd3VCxCdVwWuFuV5qJqbFYQPZoM6hd2frA8vGnl+cZ57ad 6Vcgbj/Dx3H1fjLqnUjQiGyZNbZ0WZpZBR4KFn1ilDajghUo4V3AqOUA1AQ3BlvtZNU3yB6qK jTpgPcHxxFeQyMiYEUo3C19SwBNkprLaphZoA2KG8XunuJA9Pi0vXBmiEXzJCAAn3NFZhF7TF +RbpRyg8fRWpYj9a4ZfoVaEFMtVjjhwV231vIjD82YAIFpvs2PHq4qeSw67ZTO2bpRfgSnCof MKcmpIo03ileGrhXZdg1hTNh8uYAtXdXsAWlN4x1v+AbGo9P/03CUTLpMqWN3O6PxO9ukw2Y5 WipcXxqWUuKH+CgPcsq7VTU3Ujh0OgDWHWj/Ps1s/KKsJvemZbcKw8O2JNjJ31TmJXAL7kCkA XS+ea0W1prdS3Ubyc14A8rVMx+UUlzrVGrSnMRo23axdkrYXZ0eoIIzp9Jw5yhY+jsh4QlHiY 4zvpBWKnklwzwCGL2HM8UcUm/oiN+qLrbVeCKJKjQ/U3ZQDfUPSwEWn8uKVheEdyFanXk= X-Rspamd-Queue-Id: 4Fb2Py2WS4z4WCq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.b=Vwf845fN; dmarc=pass (policy=none) header.from=gmx.net; spf=pass (mx1.freebsd.org: domain of schmiedgen@gmx.net designates 212.227.15.15 as permitted sender) smtp.mailfrom=schmiedgen@gmx.net X-Spamd-Result: default: False [-3.88 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmx.net]; R_SPF_ALLOW(-0.20)[+ip4:212.227.15.0/25]; DKIM_TRACE(0.00)[gmx.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmx.net,none]; NEURAL_HAM_SHORT(-0.88)[-0.875]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.227.15.15:from]; FREEMAIL_ENVFROM(0.00)[gmx.net]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmx.net:s=badeba3b8450]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmx.net:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[212.227.15.15:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[212.227.15.15:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.15.15:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2021 16:35:35 -0000 On 04.05.2021 21:02, Mark Johnston wrote: > This looks like fairly random kernel memory corruption. Are you able to > build an INVARIANTS kernel and test that? Assuming you're using 13.0, > you'd grab the 13.0 sources, add "options INVARIANT_SUPPORT" and > "options INVARIANTS" to the GENERIC kernel configuration in > sys/amd64/conf, and do a "make buildkernel installkernel". Below some info with an INVARIANTS kernel. Please let me know if I can pro= vide further information. Thank you! =2D-- kgdb backtrace (kgdb) backtrace #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdow= n.c:399 #2 0xffffffff80bf580b in kern_reboot (howto=3D260) at /usr/src/sys/kern/k= ern_shutdown.c:486 #3 0xffffffff80bf5c50 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80bf59b3 in panic (fmt=3D) at /usr/src/sys/kern= /kern_shutdown.c:843 #5 0xffffffff80f1ae71 in uma_dbg_free (zone=3D0xfffffe006e3e3c00, slab=3D= 0xfffff8053b159fd8, item=3D0xfffff8053b159300) at /usr/src/sys/vm/uma_core= .c:5437 #6 0xffffffff80f13a64 in item_dtor (zone=3D0xfffffe006e3e3c00, item=3D0xf= ffff8053b159300, size=3D256, udata=3D0x0, skip=3DSKIP_NONE) at /usr/src/sys/vm/uma_core.c:3220 #7 uma_zfree_arg (zone=3D0xfffffe006e3e3c00, item=3Ditem@entry=3D0xfffff8= 053b159300, udata=3Dudata@entry=3D0x0) at /usr/src/sys/vm/uma_core.c:4165 #8 0xffffffff80bcefcf in mb_free_ext (m=3Dm@entry=3D0xfffff8053b159300) a= t /usr/src/sys/kern/kern_mbuf.c:1200 #9 0xffffffff80bcda68 in m_free (m=3Dm@entry=3D0xfffff8053b159300) at /us= r/src/sys/sys/mbuf.h:1441 #10 0xffffffff80bceda8 in m_freem (mb=3Dmb@entry=3D0xfffff8053b159300) at = /usr/src/sys/kern/kern_mbuf.c:1525 #11 0xffffffff82c4d79a in div_output (so=3D, m=3D0xfffff805= 3b159300, sin=3D, control=3D) at /usr/src/sys/netinet/ip_divert.c:396 #12 div_send (so=3D, so@entry=3D, flags=3D, flags@entry=3D, m=3D0xfffff8053b159300, m@entry=3D, nam=3D, nam@entry=3D, control= =3D, control@entry=3D, td=3D, td@entry=3D) at /usr/src/sys/netinet/ip_divert.c:659 #13 0xffffffff80c92f97 in sosend_generic (so=3D0xfffff800468d5760, so@entr= y=3D, addr=3D0xfffff800120= c72e0, addr@entry=3D, uio=3D, uio@entry=3D, top=3D0xfffff8053b159300, top@entry=3D, control= =3D, control@entry=3D, flags=3D0, flags@entry=3D, td=3D0xfffffe019cdc2300, td@entry=3D) at /usr/src/sys/kern/uipc_socket.c:1755 #14 0xffffffff80c93286 in sosend (so=3D, so@entry=3D0xfffff80= 0468d5760, addr=3D, uio=3D, uio@entry=3D0xfffffe= 0199b338a8, top=3D, top@entry=3D0x0, control=3Dcontrol@entry=3D0x0, flags= =3D, flags@entry=3D0, td=3D0xfffffe019cdc2300) at /usr/src/sys/kern/uipc_socket.c:1810 #15 0xffffffff80c99ffc in kern_sendit (td=3D, td@entry=3D0x= fffffe019cdc2300, s=3D3, mp=3D, mp@entry=3D0xfffffe0199b339= 80, flags=3D0, control=3D0x0, segflg=3Dsegflg@entry=3DUIO_USERSPACE) at /usr/src/sys/kern= /uipc_syscalls.c:798 #16 0xffffffff80c9a39b in sendit (td=3D0xfffffe019cdc2300, td@entry=3D, s=3D, mp=3Dmp@entry=3D0xfffffe0199b33980, flags=3D= ) at /usr/src/sys/kern/uipc_syscalls.c:723 #17 0xffffffff80c9a1ad in sys_sendto (td=3D, td@entry=3D, uap=3D, uap@entr= y=3D) at /usr/src/sys/kern/uipc_sysca= lls.c:841 #18 0xffffffff8108824e in syscallenter (td=3D) at /usr/src/= sys/amd64/amd64/../../kern/subr_syscall.c:189 #19 amd64_syscall (td=3D0xfffffe019cdc2300, traced=3D0) at /usr/src/sys/am= d64/amd64/trap.c:1156 #20 =2D-- core.txt panic: Duplicate free of 0xfffff8053b159300 from zone 0xfffffe006e3e3c00(m= buf_packet) slab 0xfffff8053b159fd8(3) Unread portion of the kernel message buffer: <110>ipfw: 4500 Deny UDP 192.168.10.100:137 192.168.10.255:137 out via bge= 0 panic: Duplicate free of 0xfffff8053b159300 from zone 0xfffffe006e3e3c00(m= buf_packet) slab 0xfffff8053b159fd8(3) cpuid =3D 6 time =3D 1620231385 KDB: stack backtrace: #0 0xffffffff80c400e5 at kdb_backtrace+0x65 #1 0xffffffff80bf5be1 at vpanic+0x181 #2 0xffffffff80bf59b3 at panic+0x43 #3 0xffffffff80f1ae71 at uma_dbg_free+0x1e1 #4 0xffffffff80f13a64 at uma_zfree_arg+0x144 #5 0xffffffff80bcefcf at mb_free_ext+0x11f #6 0xffffffff80bcda68 at m_free+0xd8 #7 0xffffffff80bceda8 at m_freem+0x28 #8 0xffffffff82c4d79a at div_send+0x43a #9 0xffffffff80c92f97 at sosend_generic+0x5f7 #10 0xffffffff80c93286 at sosend+0x66 #11 0xffffffff80c99ffc at kern_sendit+0x1ec #12 0xffffffff80c9a39b at sendit+0x1db #13 0xffffffff80c9a1ad at sys_sendto+0x4d #14 0xffffffff8108824e at amd64_syscall+0x12e #15 0xffffffff8105bf4e at fast_syscall_common+0xf8 Uptime: 5m17s Dumping 2609 out of 65454 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.= .91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(str= uct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80bf580b in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff80bf5c50 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80bf59b3 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff80f1ae71 in uma_dbg_free (zone=3D0xfffffe006e3e3c00, slab=3D0xfffff8053b159fd8, item=3D0xfffff8053b159300) at /usr/src/sys/vm/uma_core.c:5437 #6 0xffffffff80f13a64 in item_dtor (zone=3D0xfffffe006e3e3c00, item=3D0xfffff8053b159300, size=3D256, udata=3D0x0, skip=3DSKIP_NONE) at /usr/src/sys/vm/uma_core.c:3220 #7 uma_zfree_arg (zone=3D0xfffffe006e3e3c00, item=3Ditem@entry=3D0xfffff8053b159300, udata=3Dudata@entry=3D0x0) at /usr/src/sys/vm/uma_core.c:4165 #8 0xffffffff80bcefcf in mb_free_ext (m=3Dm@entry=3D0xfffff8053b159300) at /usr/src/sys/kern/kern_mbuf.c:1200 #9 0xffffffff80bcda68 in m_free (m=3Dm@entry=3D0xfffff8053b159300) at /usr/src/sys/sys/mbuf.h:1441 #10 0xffffffff80bceda8 in m_freem (mb=3Dmb@entry=3D0xfffff8053b159300) at /usr/src/sys/kern/kern_mbuf.c:1525 #11 0xffffffff82c4d79a in div_output (so=3D, m=3D0xfffff8053b159300, sin=3D, control=3D) at /usr/src/sys/netinet/ip_divert.c:396 #12 div_send (so=3D, so@entry=3D, flags=3D, flags@entry=3D, m=3D0xfffff8053b159300, m@entry=3D, nam=3D, nam@entry=3D, control=3D, control@entry=3D, td=3D, td@entry=3D) at /usr/src/sys/netinet/ip_divert.c:659 #13 0xffffffff80c92f97 in sosend_generic (so=3D0xfffff800468d5760, so@entry=3D, addr=3D0xfffff800120c72e0, addr@entry=3D, uio=3D, uio@entry=3D, top=3D0xfffff8053b159300, top@entry=3D, control=3D, control@entry=3D, fla= gs=3D0, flags@entry=3D, td=3D0xfffffe019cdc2300, td@entry=3D) at /usr/src/sys/kern/uipc_socket.c:1755 #14 0xffffffff80c93286 in sosend (so=3D, so@entry=3D0xfffff800468d5760, addr=3D, uio=3D, uio@entry=3D0xfffffe0199b338a8, top=3D, top@entry=3D0x0, control=3Dcontrol@entry=3D0x0, flags=3D, flags@entry=3D0= , td=3D0xfffffe019cdc2300) at /usr/src/sys/kern/uipc_socket.c:1810 #15 0xffffffff80c99ffc in kern_sendit (td=3D, td@entry=3D0xfffffe019cdc2300, s=3D3, mp=3D, mp@entry=3D0xfffffe0199b33980, flags=3D0, control=3D0x0, segflg=3Dsegflg@entry=3DUIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:798 #16 0xffffffff80c9a39b in sendit (td=3D0xfffffe019cdc2300, td@entry=3D, s=3D, mp=3Dmp@entry=3D0xfffffe= 0199b33980, flags=3D) at /usr/src/sys/kern/uipc_syscalls.c:723 #17 0xffffffff80c9a1ad in sys_sendto (td=3D, td@entry=3D, uap=3D, uap@entry=3D) at /usr/src/sys/kern/uipc_syscalls.c:841 #18 0xffffffff8108824e in syscallenter (td=3D) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #19 amd64_syscall (td=3D0xfffffe019cdc2300, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1156 #20