From owner-freebsd-net@freebsd.org Wed May 5 09:08:54 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4EFDA636D2F for ; Wed, 5 May 2021 09:08:54 +0000 (UTC) (envelope-from schmiedgen@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FZrVY1pC2z3MjB; Wed, 5 May 2021 09:08:52 +0000 (UTC) (envelope-from schmiedgen@gmx.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1620205729; bh=tNBjKwIfVkZglsbFLO1l0Vrl6tEyX/iyisjOQW2K5VI=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=G9XICzuXonaSc25gHGXG5V/5e/SypIma+Lh3FmpB3P1kFJi+NaV90jVBXVeVUA5hu 5V+qDYUjSRqngfmoUiLL9lzOieO7sujvvOn94qU3hLHM4eYIUU/1AdlhJ8Tt/Qf3O5 AiQf6NwYbOvH1YaG+P2LFR1WIEUeAlnWxOIV6OWU= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.10.5] ([62.246.110.10]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MPXhA-1lqojL0qV0-00Mbir; Wed, 05 May 2021 11:08:49 +0200 Subject: Re: page fault while in kernel mode - after upgrade from 12.2 to 13.0 To: Mark Johnston Cc: freebsd-net@freebsd.org References: From: Michael Schmiedgen Message-ID: <8016b487-4256-73d1-0e61-7e245956f718@gmx.net> Date: Wed, 5 May 2021 11:08:49 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:e9Wrxlo2nTjzEOWX+/OrqAHabJ/ZqElc7tXkYg3EQnLfuH5lt8T UMQHSuPBd5FNQwVfdmAFuTs8NaMoPzS6L1MO+zV2aUqI6Xq9NE+Elo0x9y09qh3qu4pBtKG DNxEZkmSNS+BqHtlgAbZ7hkoUUoHz5UvnsGINgVqqq0kMlF8g5fXBOnizn4YarinUsBgNqi XU7M42yg5UwmvhJxQ9b2w== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:3/TWosn7UEw=:yI4gkjHtYv9C3gBOHxRyBO eX4y1whhYr3oFzYUoN62mhml4ukBueOWablKCHJ0gPxU+jIQLcZdi3W008NkauO9tvL542BqD v5fpvEqpq5K6SCoCX3GAoZyqRT5jTo1SD6NVzs1MGhWwPENMrWiNjkzGsgso98EVSvFLVfwSD OAoaLSlD0whUdqqtqL+FwhKys5LwDBF2OEDKHu05bL6EOg7WbuvsZ2JK5tdJIDDXukU0QgkG7 jeaXISek/9xW6UWlH8xYoZvjVy94jnmF+Aeuu0wEs7dDyLVWEbKYyaVir+OLzdO5XaNhmH+mB 6SbqeqdHwQeLPVCDa+eybHlXyUgJlELLoqVFfhESA819TqKAsF7uWREPXuf5Nd6OlFzqeI7/0 u6yrTSFpwd2EoYYbgqpjnh4efq4sXhDCa0cJylWGYnqGa5fbHKW76i6iLMecyY5t697EEibOO M7M47sPYW0MV26ghpdkvE7PCkU5RxcHWpJFPpdYVQs1NmyKXWJjitsD4rdntWVWvFpM6DJWA0 cTe8W052Iep5R0I/r6mIPvnGGj9bLkb3KH0f4RyxsVtUCOWDwFSn1q7BUSlAFkby3CkFdRWQw ALSu7h5rmWVdaaaPVbOm5BNPCOe25W9Jdovv+o+MDX55dOd9aVUereLNtlf3nYPvqfV6w4d3z baFgxMcU6023Lcb2wYATJx6+TyH70CrT59cAngQNJLoUxa9bwSjeHfKAaRwN+i3xInPRDfZmM gsa+zntGKUQf6nw3Wu2c3XWhVjebS17MxwF5qDry7ANgIaDpzs7tLNzJSI74Sq1m3OQk0p8oe NfvdQ3uECUKiNxTwwasX21obsfNwE7s6Ld7kcQCXYb/tGlHt/ZQJy7iAr8RpF0X3rjjbpXyFP TWBmogu4vNGr6wvnnMfwqhMqWnj+3NaulDHdsocAI3tN9tiEEm2hR/1A3sYaUQBifGDYMM9H8 +513DD5tDkU2q6p2RWwuyo7My6AY9eLVc5njppN1zdvD7NfTsUZVsatqM3miG9Cn4IMUeW71L CKCbjP9EHHknABA/hlj8Hq82hmgJz0xNPqvujm8YBHynJqpbEJ5L9Qad7IGZqYeopLWLBZn6I MZoviOFJWyeK8CVNcbkYqaVOHiUudSuk8suca+N7kx0Ck7HR7Ip0OASj0635/yAFqDnHm4RWZ OTy/ed0bwVztNjNbL1ZQ1jY2GChyPMzhf1CJbmy5G3k2rIdU5Zd707BycW46f1aoKURKU= X-Rspamd-Queue-Id: 4FZrVY1pC2z3MjB X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.b=G9XICzuX; dmarc=pass (policy=none) header.from=gmx.net; spf=pass (mx1.freebsd.org: domain of schmiedgen@gmx.net designates 212.227.15.19 as permitted sender) smtp.mailfrom=schmiedgen@gmx.net X-Spamd-Result: default: False [-4.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmx.net]; R_SPF_ALLOW(-0.20)[+ip4:212.227.15.0/25]; DKIM_TRACE(0.00)[gmx.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmx.net,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.15.19:from]; FROM_EQ_ENVFROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.227.15.19:from]; FREEMAIL_ENVFROM(0.00)[gmx.net]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmx.net:s=badeba3b8450]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmx.net:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; SPAMHAUS_ZRD(0.00)[212.227.15.19:from:127.0.2.255]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.15.19:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2021 09:08:54 -0000 On 04.05.2021 21:02, Mark Johnston wrote: > On Tue, May 04, 2021 at 08:38:39PM +0200, Michael Schmiedgen wrote: >> Hi Mark, >> >> sorry for the delay, I only can test after work. I triggered another 2 = panics, this time >> with a different result (see below). Can I provide some more informatio= n? > > This looks like fairly random kernel memory corruption. Are you able to > build an INVARIANTS kernel and test that? Assuming you're using 13.0, > you'd grab the 13.0 sources, add "options INVARIANT_SUPPORT" and > "options INVARIANTS" to the GENERIC kernel configuration in > sys/amd64/conf, and do a "make buildkernel installkernel". I will try INVARIANTS after work, but in the meantime I got 2 more panics from tonight. =2D-- #1 Fatal trap 12: page fault while in kernel mode cpuid =3D 7; apic id =3D 07 fault virtual address =3D 0x8 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80ca599c stack pointer =3D 0x28:0xfffffe0115bc46c0 frame pointer =3D 0x28:0xfffffe0115bc4700 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 12 (swi1: netisr 0) trap number =3D 12 panic: page fault cpuid =3D 7 time =3D 1620172732 KDB: stack backtrace: #0 0xffffffff80c57345 at kdb_backtrace+0x65 #1 0xffffffff80c09d21 at vpanic+0x181 #2 0xffffffff80c09b93 at panic+0x43 #3 0xffffffff8108b187 at trap_fatal+0x387 #4 0xffffffff8108b1df at trap_pfault+0x4f #5 0xffffffff8108a83d at trap+0x27d #6 0xffffffff810617a8 at calltrap+0x8 #7 0xffffffff80dbf0ae at tcp_do_segment+0x10ce #8 0xffffffff80dbd21e at tcp_input+0xabe #9 0xffffffff80dafc15 at ip_input+0x125 #10 0xffffffff80d3fa7b at swi_net+0x12b #11 0xffffffff80bcae5d at ithread_loop+0x24d #12 0xffffffff80bc7c5e at fork_exit+0x7e #13 0xffffffff8106282e at fork_trampoline+0xe Uptime: 5h36m39s Dumping 7281 out of 65454 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.= .91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(str= uct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c09916 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff80c09d90 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80c09b93 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff8108b187 in trap_fatal (frame=3D0xfffffe0115bc4600, eva=3D8) at /usr/src/sys/amd64/amd64/trap.c:915 #6 0xffffffff8108b1df in trap_pfault (frame=3Dframe@entry=3D0xfffffe0115b= c4600, usermode=3Dfalse, signo=3D, signo@entry=3D0x0, ucode=3D, ucode@entry=3D0x0) at /usr/src/sys/amd64/amd64/trap.c:732 #7 0xffffffff8108a83d in trap (frame=3D0xfffffe0115bc4600) at /usr/src/sys/amd64/amd64/trap.c:398 #8 #9 sbcut_internal (sb=3D0xfffff8043bc00610, len=3D57, len@entry=3D304) at /usr/src/sys/kern/uipc_sockbuf.c:1491 #10 0xffffffff80ca5b8a in sbcut_locked (sb=3D0xfffff8043bc00610, len=3D-1796951296, len@entry=3D304) at /usr/src/sys/kern/uipc_sockbuf= .c:1591 #11 0xffffffff80dbf0ae in tcp_do_segment (m=3D0xfffff8024b9a6900, th=3D, so=3D, tp=3D, drop_hdrlen=3D52, tlen=3D, iptos=3D0 '\000') at /usr/src/sys/netinet/tcp_input.c:2918 #12 0xffffffff80dbd21e in tcp_input (mp=3D, offp=3D, proto=3D) at /usr/src/sys/netinet/tcp_input.c:1382 #13 0xffffffff80dafc15 in ip_input (m=3D0x0) at /usr/src/sys/netinet/ip_input.c:829 #14 0xffffffff80d3fa7b in netisr_process_workstream_proto ( nwsp=3D, proto=3D1) at /usr/src/sys/net/netisr.c:919 #15 swi_net (arg=3D) at /usr/src/sys/net/netisr.c:966 #16 0xffffffff80bcae5d in intr_event_execute_handlers (p=3D= , ie=3D0xfffff80003b88c00) at /usr/src/sys/kern/kern_intr.c:1168 #17 ithread_execute_handlers (p=3D, ie=3D0xfffff80003b88c00= ) at /usr/src/sys/kern/kern_intr.c:1181 #18 ithread_loop (arg=3Darg@entry=3D0xfffff80003b95d20) at /usr/src/sys/kern/kern_intr.c:1269 #19 0xffffffff80bc7c5e in fork_exit ( callout=3D0xffffffff80bcac10 , arg=3D0xfffff80003b95d20= , frame=3D0xfffffe0115bc4b00) at /usr/src/sys/kern/kern_fork.c:1069 =2D-- #2 Unread portion of the kernel message buffer: panic: sbappendaddr_locked cpuid =3D 2 time =3D 1620181490 KDB: stack backtrace: #0 0xffffffff80c57345 at kdb_backtrace+0x65 #1 0xffffffff80c09d21 at vpanic+0x181 #2 0xffffffff80c09b93 at panic+0x43 #3 0xffffffff80ca51e0 at sbappendaddr_locked_internal+0 #4 0xffffffff82c4efd0 at divert_packet+0x1a0 #5 0xffffffff82c2bc81 at ipfw_check_packet+0x2c1 #6 0xffffffff80d41f87 at pfil_run_hooks+0x97 #7 0xffffffff80dafeb5 at ip_input+0x3c5 #8 0xffffffff80d3f2da at netisr_dispatch_src+0xca #9 0xffffffff80d23a68 at ether_demux+0x148 #10 0xffffffff80d24dec at ether_nh_input+0x34c #11 0xffffffff80d3f2da at netisr_dispatch_src+0xca #12 0xffffffff80d23eb9 at ether_input+0x69 #13 0xffffffff80d2074a at if_input+0xa #14 0xffffffff8060a98e at bge_rxeof+0x49e #15 0xffffffff80607f27 at bge_intr_task+0x1a7 #16 0xffffffff80c6afe1 at taskqueue_run_locked+0x181 #17 0xffffffff80c6c2fc at taskqueue_thread_loop+0xac Uptime: 2h21m11s Dumping 8148 out of 65454 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.= .91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(str= uct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c09916 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff80c09d90 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80c09b93 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff80ca51e0 in sbappendaddr_locked (sb=3D0xfffff8002829a8a8, asa=3D0xfffffe0115ebc5a0, m0=3D0xfffff804a977b700, control=3D0x0) at /usr/src/sys/kern/uipc_sockbuf.c:1198 #6 0xffffffff82c4efd0 in divert_packet (m=3D0xfffff804a977b700, incoming=3D) at /usr/src/sys/netinet/ip_divert.c:285 #7 0xffffffff82c2bc81 in ipfw_divert (m0=3D0xfffffe0115ebc760, args=3D0xfffffe0115ebc610, tee=3D) at /usr/src/sys/netpfil/ipfw/ip_fw_pfil.c:525 #8 ipfw_check_packet (m0=3D0xfffffe0115ebc760, ifp=3D0xfffff8000506f000, flags=3D65536, ruleset=3D, inp=3D0x0) at /usr/src/sys/netpfil/ipfw/ip_fw_pfil.c:283 #9 0xffffffff80d41f87 in pfil_run_hooks (head=3D, p=3D..., ifp=3Difp@entry=3D0xfffff8000506f000, flags=3Dflags@entry=3D65536, inp=3Dinp@entry=3D0x0) at /usr/src/sys/net/pfil.c:187 #10 0xffffffff80dafeb5 in ip_input (m=3D0x0) at /usr/src/sys/netinet/ip_input.c:610 #11 0xffffffff80d3f2da in netisr_dispatch_src (proto=3D1, source=3D, source@entry=3D0, m=3D) at /usr/src/sys/net/netisr.c:1143 #12 0xffffffff80d3f5cf in netisr_dispatch (proto=3D, m=3D) at /usr/src/sys/net/netisr.c:1234 #13 0xffffffff80d23a68 in ether_demux (ifp=3Difp@entry=3D0xfffff8000506f00= 0, m=3D) at /usr/src/sys/net/if_ethersubr.c:923 #14 0xffffffff80d24dec in ether_input_internal (ifp=3D0xfffff8000506f000, m=3D) at /usr/src/sys/net/if_ethersubr.c:709 #15 ether_nh_input (m=3D) at /usr/src/sys/net/if_ethersubr.= c:739 #16 0xffffffff80d3f2da in netisr_dispatch_src (proto=3Dproto@entry=3D5, source=3D, source@entry=3D0, m=3D, m@entry=3D0xfffff804a977b700) at /usr/src/sys/net/netisr.c:1143 #17 0xffffffff80d3f5cf in netisr_dispatch (proto=3D, proto@entry=3D5, m=3D, m@entry=3D0xfffff804a977b700) at /usr/src/sys/net/netisr.c:1234 #18 0xffffffff80d23eb9 in ether_input (ifp=3D, ifp@entry=3D, m=3D0xfffff804a977b700, m@entry=3D) at /usr/src/sys/net/if_ethersubr.c:830 #19 0xffffffff80d2074a in if_input (ifp=3D, ifp@entry=3D0xfffff8000506f000, sendmp=3D, sendmp@entry=3D0xfffff804a977b700) at /usr/src/sys/net/if.c:4391 #20 0xffffffff8060a98e in bge_rxeof (sc=3Dsc@entry=3D0xfffffe0115cd4000, rx_prod=3Drx_prod@entry=3D448, holdlck=3Dholdlck@entry=3D0) at /usr/src/sys/dev/bge/if_bge.c:4412 #21 0xffffffff80607f27 in bge_intr_task (arg=3D0xfffffe0115cd4000, pending=3D) at /usr/src/sys/dev/bge/if_bge.c:4642 #22 0xffffffff80c6afe1 in taskqueue_run_locked ( queue=3Dqueue@entry=3D0xfffff80005051d00) at /usr/src/sys/kern/subr_taskqueue.c:476 #23 0xffffffff80c6c2fc in taskqueue_thread_loop (arg=3D, arg@entry=3D0xfffffe0115cdb568) at /usr/src/sys/kern/subr_taskqueue.c= :793 #24 0xffffffff80bc7c5e in fork_exit ( callout=3D0xffffffff80c6c250 , arg=3D0xfffffe0115cdb568, frame=3D0xfffffe0115ebcb00) at /usr/src/sys/kern/kern_fork.c:1069