From owner-freebsd-stable@freebsd.org Sat Oct 20 20:18:52 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94C96FF4115 for ; Sat, 20 Oct 2018 20:18:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2E32E860C6 for ; Sat, 20 Oct 2018 20:18:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E4115FF4113; Sat, 20 Oct 2018 20:18:51 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B188CFF4111 for ; Sat, 20 Oct 2018 20:18:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49A84860C4 for ; Sat, 20 Oct 2018 20:18:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 75BE41B839 for ; Sat, 20 Oct 2018 20:18:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w9KKIoKB094276 for ; Sat, 20 Oct 2018 20:18:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w9KKIokS094275 for stable@FreeBSD.org; Sat, 20 Oct 2018 20:18:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: stable@FreeBSD.org Subject: [Bug 227654] [panic] repeatable crash with lagg+vlan+em Date: Sat, 20 Oct 2018 20:18:50 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: eugen@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2018 20:18:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227654 --- Comment #4 from Eugene Grosbein --- This is 100% repeatable using same command sequence under 12.0-BETA1/i386 installed with all defaults inside VirtualBox VM. This time it says: panic: vm_fault_hold: fault on nofault entry, addr: 0 It generates nice crashdump and reboots. I've uploaded kernel.debug (stock one from 12.0-BETA1/i386 installation ima= ge, 18M compressed) and vmcore.0.xz (9.2MB compressed) here: http://www.grosbein.net/freebsd/crash/20181021/ Here comes kgdb script: Script started on Sun Oct 21 02:31:58 2018 Command: kgdb kernel.debug /var/crash/vmcore.0 GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD] Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kernel.debug...done. Unread portion of the kernel message buffer: <6>em0: link state changed to DOWN <6>vlan61: link state changed to DOWN panic: vm_fault_hold: fault on nofault entry, addr: 0 cpuid =3D 0 time =3D 1540062884 KDB: stack backtrace: #0 0x112d01f at kdb_backtrace+0x4f #1 0x10e03f7 at vpanic+0x147 #2 0x10e02ab at panic+0x1b #3 0x14289c5 at vm_fault_hold+0x2a45 #4 0x1425f2e at vm_fault+0x5e #5 0x16b6ef7 at trap_pfault+0xc7 #6 0x16b64af at trap+0x3cf #7 0xffc0315d at PTDpde+0x4165 #8 0x11e0122 at ether_output+0x6a2 #9 0x124887d at arprequest+0x44d #10 0x12493f9 at arp_ifinit+0x59 #11 0x124c5bb at arp_handle_ifllchange+0x3b #12 0x11db275 at if_setlladdr+0x275 #13 0x11eb900 at vlan_lladdr_fn+0x30 #14 0x113eba9 at taskqueue_run_locked+0x189 #15 0x113fd57 at taskqueue_thread_loop+0x97 #16 0x10a1af1 at fork_exit+0x71 #17 0xffc033ba at PTDpde+0x43c2 Uptime: 7m21s Physical memory: 2019 MB Dumping 84 MB: 69 53 37 21 5 __curthread () at ./machine/pcpu.h:226 226 ./machine/pcpu.h: No such file or directory. (kgdb) add-kld if_tap.ko add symbol table from file "if_tap.ko.debug" at .rodata_addr =3D 0x18c0c134 set_sysctl_set_addr =3D 0x18c0c4e0 set_modmetadata_set_addr =3D 0x18c0c4f8 .note.gnu.build-id_addr =3D 0x18c0c500 .dynsym_addr =3D 0x18c0c548 .gnu.hash_addr =3D 0x18c0cb18 .hash_addr =3D 0x18c0cb58 .dynstr_addr =3D 0x18c0ce48 .rel.dyn_addr =3D 0x18c0d348 .text_addr =3D 0x18c0f000 .data_addr =3D 0x18c12000 set_sysinit_set_addr =3D 0x18c12270 set_sysuninit_set_addr =3D 0x18c12278 .dynamic_addr =3D 0x18c13000 .bss_addr =3D 0x18c14000 (y or n) y Reading symbols from if_tap.ko.debug...done. (kgdb) add-kld if_lagg.ko add symbol table from file "if_lagg.ko.debug" at .rodata_addr =3D 0x18c15138 set_sysctl_set_addr =3D 0x18c16038 set_modmetadata_set_addr =3D 0x18c16054 .note.gnu.build-id_addr =3D 0x18c16060 .dynsym_addr =3D 0x18c160a8 .gnu.hash_addr =3D 0x18c16798 .hash_addr =3D 0x18c167e8 .dynstr_addr =3D 0x18c16b68 .rel.dyn_addr =3D 0x18c171e8 .text_addr =3D 0x18c1a000 .data_addr =3D 0x18c24000 set_vnet_addr =3D 0x18c24270 set_sysinit_set_addr =3D 0x18c242a0 set_sysuninit_set_addr =3D 0x18c242ac .dynamic_addr =3D 0x18c25000 .bss_addr =3D 0x18c26000 (y or n) y Reading symbols from if_lagg.ko.debug...done. (kgdb) bt full #0 __curthread () at ./machine/pcpu.h:226 td =3D #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:366 error =3D coredump =3D #2 0x010e0073 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:446 once =3D #3 0x010e0444 in vpanic ( fmt=3D0x179ab2c "%s: fault on nofault entry, addr: %#lx",=20 ap=3D0x16a1462c "U\327p\001") at /usr/src/sys/kern/kern_shutdown.c:872 buf =3D "vm_fault_hold: fault on nofault entry, addr: 0", '\000' td =3D 0x7369700 newpanic =3D bootopt =3D 260 other_cpus =3D #4 0x010e02ab in panic ( fmt=3D0x179ab2c "%s: fault on nofault entry, addr: %#lx") at /usr/src/sys/kern/kern_shutdown.c:799 ap =3D --Type for more, q to quit, c to continue without paging--c #5 0x014289c5 in vm_fault_hold (map=3D0x2bd5000, vaddr=3D0, fault_type=3D<= optimized out>, fault_flags=3D0, m_hold=3D0x0) at /usr/src/sys/vm/vm_fault.c:586 hardfault =3D nera =3D faultcount =3D wired =3D 0 prot =3D 7 '\a' result =3D 0 rv =3D behind =3D ahead =3D error =3D locked =3D vp =3D fs =3D dset =3D next_object =3D alloc_req =3D era =3D behavior =3D cluster_offset =3D e_start =3D e_end =3D is_first_object_locked =3D retry_prot =3D retry_pindex =3D retry_object =3D dead =3D #6 0x01425f2e in vm_fault (map=3D0x2bd5000, vaddr=3D0, fault_type=3D1 '\00= 1', fault_flags=3D0) at /usr/src/sys/vm/vm_fault.c:536 td =3D 0x7369700 result =3D #7 0x016b6ef7 in trap_pfault (frame=3D0x16a14894, usermode=3D0, eva=3D28) = at /usr/src/sys/i386/i386/trap.c:882 td =3D 0x7369700 p =3D va =3D ftype =3D rv =3D map =3D #8 0x016b64af in trap (frame=3D0x16a14894) at /usr/src/sys/i386/i386/trap.= c:519 td =3D dr6 =3D addr =3D ucode =3D signo =3D p =3D 0x206c1e4 type =3D 12 eva =3D 28 ksi =3D #9 0xffc0315d in ?? () No symbol table info available. #10 0x16a14894 in ?? () No symbol table info available. #11 0x011e0122 in ether_output (ifp=3D0x7240400, m=3D, dst=3D0= x16a149e8, ro=3D0x16a149a8) at /usr/src/sys/net/if_ethersubr.c:435 linkhdr =3D "\001\000\000\000\000\227\066\a\\I\241\026\372\251" error =3D pflags =3D 0 addref =3D 0 phdr =3D hlen =3D lle =3D eh =3D t =3D #12 0x0124887d in arprequest (ifp=3D0x7240400, sip=3D0x16a14a40, tip=3D0x16= a14a40, enaddr=3D0x1911d94a "") at /usr/src/sys/netinet/if_ether.c:428 linkhdr =3D "\377\377\377\377\377\377\000\275\024\247\377\000\b\006\000\000\000\000\000= \000\000\000\000" carpaddr =3D m =3D ah =3D linkhdrsize =3D error =3D 0 ro =3D {ro_rt =3D 0x0, ro_lle =3D 0x0, ro_prepend =3D 0x16a149d0 "\377\377\377\377\377\377", ro_plen =3D 14, ro_flags =3D 0, ro_mtu =3D 0, s= pare =3D 0, ro_dst =3D {sa_len =3D 0 '\000', sa_family =3D 0 '\000', sa_data =3D '\000'= }} sa =3D {sa_len =3D 2 '\002', sa_family =3D 35 '#', sa_data =3D "\000\000J\331\021\031\000\000\000\000\000\227\066\a"} #13 0x012493f9 in arp_announce_ifaddr (ifp=3D0x7240400, addr=3D..., enaddr=3D) at /usr/src/sys/netinet/if_ether.c:1436 No locals. #14 arp_ifinit (ifp=3D0x7240400, ifa=3D0x1911d200) at /usr/src/sys/netinet/if_ether.c:1423 dst_in =3D 0x1911d250 dst =3D 0x1911d250 #15 0x0124c5bb in arp_handle_ifllchange (ifp=3D0x7240400) at /usr/src/sys/netinet/if_ether.c:1450 ifa =3D 0x1911d200 #16 0x011db275 in if_setlladdr (ifp=3D0x7240400, lladdr=3D0x1911d94a "", le= n=3D6) at /usr/src/sys/net/if.c:3867 _ep =3D _t =3D _el =3D ifa =3D sdl =3D nep_et =3D {datap =3D {0x0, 0x16a26c80, 0xdeadbeef}, datai =3D {1}} ifr =3D #17 0x011eb900 in vlan_lladdr_fn (arg=3D0x1911ab00, pending=3D1) at /usr/src/sys/net/if_vlan.c:1306 ifv =3D 0x1911ab00 ifp =3D #18 0x0113eba9 in taskqueue_run_locked (queue=3D0x736d600) at /usr/src/sys/kern/subr_taskqueue.c:465 tb_first =3D pending =3D 1 task =3D 0x1911ab28 tb =3D #19 0x0113fd57 in taskqueue_thread_loop (arg=3D0x206f918 = ) at /usr/src/sys/kern/subr_taskqueue.c:757 tqp =3D tq =3D 0x736d600 #20 0x010a1af1 in fork_exit (callout=3D0x113fcc0 , arg=3D0x206f918 , frame=3D0x16a14ba8) at /usr/src/sys/kern/kern_fork.c:1057 td =3D 0x7369700 p =3D 0x206c1e4 dtd =3D #21 0xffc033ba in ?? () No symbol table info available. (kgdb) frame 11 #11 0x011e0122 in ether_output (ifp=3D0x7240400, m=3D,=20 dst=3D0x16a149e8, ro=3D0x16a149a8) at /usr/src/sys/net/if_ethersubr.c:4= 35 435 return ether_output_frame(ifp, m); (kgdb) p *ifp $1 =3D {if_link =3D {cstqe_next =3D 0x0}, if_clones =3D {le_next =3D 0x0,=20 le_prev =3D 0x757ff1c}, if_groups =3D {cstqh_first =3D 0x757ec70,=20 cstqh_last =3D 0x756d1a4}, if_alloctype =3D 6 '\006', if_softc =3D 0x19= 11ab00,=20 if_llsoftc =3D 0x0, if_l2com =3D 0x0, if_dname =3D 0x1b08e5c "= vlan",=20 if_dunit =3D 61, if_index =3D 6, if_index_reserved =3D 0,=20 if_xname =3D "vlan61\000\000\000\000\000\000\000\000\000",=20 if_description =3D 0x0, if_flags =3D 34819, if_drv_flags =3D 64,=20 if_capabilities =3D 3, if_capenable =3D 3, if_linkmib =3D 0x1911ab14,=20 if_linkmiblen =3D 20, if_refcount =3D 1, if_type =3D 135 '\207',=20 if_addrlen =3D 6 '\006', if_hdrlen =3D 4 '\004', if_link_state =3D 1 '\00= 1',=20 if_mtu =3D 1500, if_metric =3D 0, if_baudrate =3D 0, if_hwassist =3D 6,=20 if_epoch =3D 421, if_lastchange =3D {tv_sec =3D 1540062864, tv_usec =3D 9= 21826},=20 if_snd =3D {ifq_head =3D 0x0, ifq_tail =3D 0x0, ifq_len =3D 0, ifq_maxlen= =3D 50,=20 ifq_mtx =3D {lock_object =3D {lo_name =3D 0x7240430 "vlan61",=20 lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock= =3D 0},=20 ifq_drv_head =3D 0x0, ifq_drv_tail =3D 0x0, ifq_drv_len =3D 0,=20 ifq_drv_maxlen =3D 0, altq_type =3D 0, altq_flags =3D 0, altq_disc =3D = 0x0,=20 altq_ifp =3D 0x7240400, altq_enqueue =3D 0x0, altq_dequeue =3D 0x0,=20 altq_request =3D 0x0, altq_clfier =3D 0x0, altq_classify =3D 0x0,=20 altq_tbr =3D 0x0, altq_cdnr =3D 0x0}, if_linktask =3D {ta_link =3D { stqe_next =3D 0x0}, ta_pending =3D 0, ta_priority =3D 0,=20 ta_func =3D 0x11d5460 , ta_context =3D 0x7240400}= ,=20 if_addr_lock =3D {lock_object =3D {lo_name =3D 0x1704959 "if_addr_lock",= =20 lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock = =3D 0},=20 --Type for more, q to quit, c to continue without paging-- c if_addrhead =3D {cstqh_first =3D 0x1911d900, cstqh_last =3D 0x1911d214}, if_multiaddrs =3D {cstqh_first =3D 0x1911ac80, cstqh_last =3D 0x1911a0c0}, = if_amcount =3D 0, if_addr =3D 0x1911d900, if_hw_addr =3D 0x757ebd0, if_broadcastaddr = =3D 0x1b08424 "\377\377\377\377\377\377", if_afdata_lock =3D {lock_o= bject =3D {lo_name =3D 0x1771f6b "if_afdata", lo_flags =3D 16973824, lo_data =3D = 0, lo_witness =3D 0x0}, mtx_lock =3D 0}, if_afdata =3D {0x0, 0x0, 0x757ece0, 0= x0 , 0x74cff80, 0x0 }, if_afdata_initializ= ed =3D 2, if_fib =3D 0, if_vnet =3D 0x7017060, if_home_vnet =3D 0x7017060, if_vlan= trunk =3D 0x0, if_bpf =3D 0x1911aa80, if_pcount =3D 0, if_bridge =3D 0x0, if_lagg =3D= 0x0, if_pf_kif =3D 0x0, if_carp =3D 0x0, if_label =3D 0x0, if_netmap =3D 0x0, if= _output =3D 0x11dfa80 , if_input =3D 0x11e0720 , if_bridge_i= nput =3D 0x0, if_bridge_output =3D 0x0, if_bridge_linkstate =3D 0x0, if_start =3D 0x= 0, if_ioctl =3D 0x11ea700 , if_init =3D 0x11ea470 , if_resolvemulti =3D 0x11e0780 , if_qflush =3D 0x11ea6f0 , if_transmit =3D 0x11ea480 , if_reassign =3D 0= x11e0980 , if_get_counter =3D 0x11d5720 , if_requestencap =3D 0x11e08b0 , if_counters =3D {0x16b6= 1000, 0x16b61010, 0x16b61020, 0x16b61030, 0x16b61040, 0x16b61050, 0x16b61060, 0x16b61070, 0x16b61080, 0x16b61090, 0x16b610a0, 0x16b610b0}, if_hw_tsomax = =3D 65518, if_hw_tsomaxsegcount =3D 35, if_hw_tsomaxsegsize =3D 2048, if_snd_ta= g_alloc =3D 0x0, if_snd_tag_modify =3D 0x0, if_snd_tag_query =3D 0x0, if_snd_tag_fr= ee =3D 0x0, if_pcp =3D 0 '\000', if_netdump_methods =3D 0x0, if_epoch_ctx =3D {data =3D= {0x0, 0x0}}, if_addr_et =3D {datap =3D {0x0, 0x0, 0x0}, datai =3D {0}}, if_maddr_= et =3D {datap =3D {0x0, 0x0, 0x0}, datai =3D {0}}, if_ispare =3D {0, 0, 0, 0}} (kgdb) p m $2 =3D (kgdb) l 430 if (m =3D=3D NULL) 431 return (0); 432 } 433=20=20=20=20=20 434 /* Continue with link-layer output */ 435 return ether_output_frame(ifp, m); 436 } 437=20=20=20=20=20 438 static bool 439 ether_set_pcp(struct mbuf **mp, struct ifnet *ifp, uint8_t pcp) (kgdb) p ifp->if_type $4 =3D 135 '\207' (kgdb) p ifp->if_transmit $5 =3D (if_transmit_fn_t) 0x11ea480 (kgdb) p *((struct ifvlan *)ifp->if_softc)->ifv_trunk->parent $8 =3D {if_link =3D {cstqe_next =3D 0x757d000}, if_clones =3D {le_next =3D = 0x0,=20 le_prev =3D 0x0}, if_groups =3D {cstqh_first =3D 0x74fe980,=20 cstqh_last =3D 0x74fe984}, if_alloctype =3D 6 '\006', if_softc =3D 0x75= 2a000,=20 if_llsoftc =3D 0x0, if_l2com =3D 0x0, if_dname =3D 0x70444b0 "em", if_dun= it =3D 0,=20 if_index =3D 1, if_index_reserved =3D 0,=20 if_xname =3D "em0", '\000' , if_description =3D 0x0,=20 if_flags =3D 34819, if_drv_flags =3D 64, if_capabilities =3D 8468635,=20 if_capenable =3D 8454299, if_linkmib =3D 0x0, if_linkmiblen =3D 0,=20 if_refcount =3D 4, if_type =3D 161 '\241', if_addrlen =3D 6 '\006',=20 if_hdrlen =3D 18 '\022', if_link_state =3D 1 '\001', if_mtu =3D 1500,=20 if_metric =3D 0, if_baudrate =3D 0, if_hwassist =3D 6, if_epoch =3D 1,=20 if_lastchange =3D {tv_sec =3D 1540062449, tv_usec =3D 317821}, if_snd =3D= { ifq_head =3D 0x0, ifq_tail =3D 0x0, ifq_len =3D 0, ifq_maxlen =3D 1023,= ifq_mtx =3D { lock_object =3D {lo_name =3D 0x752b430 "em0", lo_flags =3D 16973824,= =20 lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}, ifq_drv_head = =3D 0x0,=20 ifq_drv_tail =3D 0x0, ifq_drv_len =3D 0, ifq_drv_maxlen =3D 1023, altq_= type =3D 0,=20 altq_flags =3D 1, altq_disc =3D 0x0, altq_ifp =3D 0x752b400, altq_enque= ue =3D 0x0,=20 altq_dequeue =3D 0x0, altq_request =3D 0x0, altq_clfier =3D 0x0,=20 altq_classify =3D 0x0, altq_tbr =3D 0x0, altq_cdnr =3D 0x0}, if_linktas= k =3D { ta_link =3D {stqe_next =3D 0x0}, ta_pending =3D 0, ta_priority =3D 0,=20 ta_func =3D 0x11d5460 , ta_context =3D 0x752b400}= ,=20 if_addr_lock =3D {lock_object =3D {lo_name =3D 0x1704959 "if_addr_lock",= =20 lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0x0},=20 mtx_lock =3D 129987456}, if_addrhead =3D {cstqh_first =3D 0x74e0c00,=20 --Type for more, q to quit, c to continue without paging--q cstqh_last =3D Quit Command exit status: 0 Script done on Sun Oct 21 02:55:39 2018 --=20 You are receiving this mail because: You are on the CC list for the bug.=