From owner-freebsd-net@freebsd.org Fri Dec 14 23:51:43 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8483E1324D47 for ; Fri, 14 Dec 2018 23:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1BB017114E for ; Fri, 14 Dec 2018 23:51:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id D38301324D46; Fri, 14 Dec 2018 23:51:42 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE2B41324D44 for ; Fri, 14 Dec 2018 23:51:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 24E697114B for ; Fri, 14 Dec 2018 23:51:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 68EA060EB for ; Fri, 14 Dec 2018 23:51:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBENpfeV034742 for ; Fri, 14 Dec 2018 23:51:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBENpfS9034741 for net@FreeBSD.org; Fri, 14 Dec 2018 23:51:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 234026] [panic] [dummynet] Repeatable panic in dummynet due to locking issues and use-after-free Date: Fri, 14 Dec 2018 23:51:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: eugen@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status keywords bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2018 23:51:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234026 Bug ID: 234026 Summary: [panic] [dummynet] Repeatable panic in dummynet due to locking issues and use-after-free Product: Base System Version: 11.2-STABLE Hardware: Any OS: Any Status: New Keywords: crash Severity: Affects Some People Priority: --- Component: kern Assignee: net@FreeBSD.org Reporter: eugen@freebsd.org Hi! I run multiple routers using FreeBSD 11.2-STABLE/amd64 r336962, ipfw+dummyn= et and net/mpd5 daemon that dynamically creates/destroys ngXXX interfaces for multiple PPPoE clients. If an interface ngXXX is destroyed while dummynet pipe/queue keeps mbuf with m_pkthdr.rcvif pointing to freed struct ifnet, kernel panices when taskqueue runs dummynet_task/dummynet_send/netisr_dispatch_src/ip_input sequence and I have crashdump. kgdb session follows: Script started on Sat Dec 15 06:47:49 2018 Command: kgdb kernel.debug /home/nanobsd/pppoe/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: stack pointer =3D 0x28:0xfffffe01244bb920 frame pointer =3D 0x28:0xfffffe01244bb9a0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 0 (dummynet) trap number =3D 12 panic: page fault cpuid =3D 0 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff802fc89b =3D db_trace_self_wrapper+0x2b/frame 0xfffffe01244bb5d0 vpanic() at 0xffffffff804f0ac7 =3D vpanic+0x177/frame 0xfffffe01244bb630 panic() at 0xffffffff804f0943 =3D panic+0x43/frame 0xfffffe01244bb690 trap_fatal() at 0xffffffff8076f2af =3D trap_fatal+0x35f/frame 0xfffffe01244= bb6e0 trap_pfault() at 0xffffffff8076f309 =3D trap_pfault+0x49/frame 0xfffffe0124= 4bb740 trap() at 0xffffffff8076eae4 =3D trap+0x2d4/frame 0xfffffe01244bb850 calltrap() at 0xffffffff8074ff3c =3D calltrap+0x8/frame 0xfffffe01244bb850 --- trap 0xc, rip =3D 0xffffffff804ec893, rsp =3D 0xfffffe01244bb920, rbp = =3D 0xfffffe01244bb9a0 --- __rw_rlock_hard() at 0xffffffff804ec893 =3D __rw_rlock_hard+0xf3/frame 0xfffffe01244bb9a0 ip_input() at 0xffffffff806444ca =3D ip_input+0x53a/frame 0xfffffe01244bba30 netisr_dispatch_src() at 0xffffffff8060ebe8 =3D netisr_dispatch_src+0xa8/fr= ame 0xfffffe01244bba80 dummynet_send() at 0xffffffff806723dd =3D dummynet_send+0x10d/frame 0xfffffe01244bbab0 dummynet_task() at 0xffffffff80671e1c =3D dummynet_task+0x2ec/frame 0xfffffe01244bbb20 taskqueue_run_locked() at 0xffffffff80548a54 =3D taskqueue_run_locked+0x154= /frame 0xfffffe01244bbb80 taskqueue_thread_loop() at 0xffffffff80549bb8 =3D taskqueue_thread_loop+0x98/frame 0xfffffe01244bbbb0 fork_exit() at 0xffffffff804ba803 =3D fork_exit+0x83/frame 0xfffffe01244bbb= f0 fork_trampoline() at 0xffffffff80750eee =3D fork_trampoline+0xe/frame 0xfffffe01244bbbf0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- Uptime: 57d17h28m40s Dumping 467 out of 4073 MB:..4%..11%..21%..31%..42%..52%..62%..72%..83%..93% Reading symbols from /boot/modules/tmpfs.ko...done. Loaded symbols for /boot/modules/tmpfs.ko #0 doadump (textdump=3D1) at pcpu.h:230 230 __asm("movq %%gs:%1,%0" : "=3Dr" (td) (kgdb) bt #0 doadump (textdump=3D1) at pcpu.h:230 #1 0xffffffff804f06c0 in kern_reboot (howto=3D260) at /home/src/sys/kern/kern_shutdown.c:383 #2 0xffffffff804f0b01 in vpanic (fmt=3D, ap=3D) at /home/src/sys/kern/kern_shutdown.c:776 #3 0xffffffff804f0943 in panic (fmt=3D) at /home/src/sys/kern/kern_shutdown.c:707 #4 0xffffffff8076f2af in trap_fatal (frame=3D0xfffffe01244bb860, eva=3D274877908504) at /home/src/sys/amd64/amd64/trap.c:877 #5 0xffffffff8076f309 in trap_pfault (frame=3D0xfffffe01244bb860, usermode= =3D0) at pcpu.h:230 #6 0xffffffff8076eae4 in trap (frame=3D0xfffffe01244bb860) at /home/src/sys/amd64/amd64/trap.c:415 #7 0xffffffff8074ff3c in calltrap () at /home/src/sys/amd64/amd64/exception.S:231 #8 0xffffffff804ec893 in __rw_rlock_hard (rw=3D0xfffff80092e78190, td=3D0xfffff80001d02620, v=3D) at /home/src/sys/kern/kern_rwlock.c:493 #9 0xffffffff806444ca in ip_input (m=3D) at /home/src/sys/netinet/ip_input.c:795 #10 0xffffffff8060ebe8 in netisr_dispatch_src (proto=3D1, source=3D, m=3D) at /home/src/sys/net/netisr.c:1120 #11 0xffffffff806723dd in dummynet_send (m=3D0x0) at /home/src/sys/netpfil/ipfw/ip_dn_io.c:774 #12 0xffffffff80671e1c in dummynet_task (context=3D, pending=3D) at /home/src/sys/netpfil/ipfw/ip_dn_io= .c:729 #13 0xffffffff80548a54 in taskqueue_run_locked (queue=3D0xfffff80006085e00) at /home/src/sys/kern/subr_taskqueue.c:463 #14 0xffffffff80549bb8 in taskqueue_thread_loop (arg=3D) at /home/src/sys/kern/subr_taskqueue.c:755 #15 0xffffffff804ba803 in fork_exit (callout=3D0xffffffff80549b20 , arg=3D0xffffffff80c82c38, frame=3D0xfffffe01244bbc00) at /home/src/sys/kern/kern_fork.c:1072 #16 0xffffffff80750eee in fork_trampoline () at /home/src/sys/amd64/amd64/exception.S:972 ---Type to continue, or q to quit--- #17 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) frame 9 #9 0xffffffff806444ca in ip_input (m=3D) at /home/src/sys/netinet/ip_input.c:795 795 IF_ADDR_RLOCK(ifp); (kgdb) l 790 * interface. Reception of forwarded directed broadcasts w= ould 791 * be handled via ip_forward() and ether_output() with the loopback 792 * into the stack for SIMPLEX interfaces handled by ether_output(). 793 */ 794 if (ifp !=3D NULL && ifp->if_flags & IFF_BROADCAST) { 795 IF_ADDR_RLOCK(ifp); 796 TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { 797 if (ifa->ifa_addr->sa_family !=3D AF_INET) 798 continue; 799 ia =3D ifatoia(ifa); (kgdb) p *ifp $1 =3D {if_link =3D {tqe_next =3D 0x4000000004, tqe_prev =3D 0x4000000006},= if_clones =3D { le_next =3D 0x4000000007, le_prev =3D 0x4000000009}, if_groups =3D {tqh= _first =3D 0x400000000a, tqh_last =3D 0x4000000011}, if_alloctype =3D 250 '=D0=97', if_softc =3D= 0x4000000104, if_llsoftc =3D 0x40000004d0, if_l2com =3D 0x40000004d4, if_dname =3D 0x4000000184
, if_dunit = =3D 218, if_index =3D 64, if_index_reserved =3D 0, if_xname =3D 0xfffff80092e78060 "\220\001", if_description =3D 0x400000035e
, if_= flags =3D 1050, if_drv_flags =3D 64, if_capabilities =3D 454, if_capenable =3D 64, if_lin= kmib =3D 0x4000000386, if_linkmiblen =3D 274877907462, if_refcount =3D 682, if_type =3D 64 '@', = if_addrlen =3D 0 '\0', if_hdrlen =3D 0 '\0', if_link_state =3D 0 '\0', if_mtu =3D 522, if_metric= =3D 64, if_baudrate =3D 274877907476, if_hwassist =3D 274877907488, if_epoch =3D 274877907500, if_lastchange =3D {tv_sec =3D 274877908294, tv_usec =3D 274877907730}, if= _snd =3D { ifq_head =3D 0x40000002e0, ifq_tail =3D 0x4000000334, ifq_len =3D 824, = ifq_maxlen =3D 64, ifq_mtx =3D { lock_object =3D {lo_name =3D 0x40000003c6
, lo_flags =3D 1298, lo_data =3D 64, lo_witness =3D 0x4000000332}, mt= x_lock =3D 274877907950}, ifq_drv_head =3D 0x40000002ae, ifq_drv_tail =3D 0x40000000fc, ifq_drv_l= en =3D 858, ifq_drv_maxlen =3D 64, altq_type =3D 870, altq_flags =3D 64, altq_disc = =3D 0x400000036a, altq_ifp =3D 0x4000000124, altq_enqueue =3D 0x4000000318, altq_dequeue = =3D 0x400000030a, altq_request =3D 0x400000036c, altq_clfier =3D 0x4000000188, altq_class= ify =3D 0x400000058d, altq_tbr =3D 0x400000058f, altq_cdnr =3D 0x4000000376}, if_linktask =3D= {ta_link =3D { stqe_next =3D 0x4000000262}, ta_pending =3D 460, ta_priority =3D 0, t= a_func =3D 0x4000000264, ta_context =3D 0x40000001b6}, if_addr_lock =3D {lock_object =3D { lo_name =3D 0x40000001b8
, lo_fla= gs =3D 1072, lo_data =3D 64, lo_witness =3D 0x400000026a}, rw_lock =3D 274877907356}, if_addrhead = =3D { tqh_first =3D 0x4000000382, tqh_last =3D 0x4000000196}, if_multiaddrs = =3D { tqh_first =3D 0x4000000120, tqh_last =3D 0x4000000218}, if_amcount =3D = 294, if_addr =3D 0x40000001be, if_broadcastaddr =3D 0x4000000064
, if_afdata_lock =3D { ---Type to continue, or q to quit--- lock_object =3D {lo_name =3D 0x4000000192
, lo_flags =3D 810, lo_data =3D 64, lo_witness =3D 0x40000002de}, rw_lock =3D 27487790768= 4}, if_afdata =3D 0xfffff80092e78208, if_afdata_initialized =3D 441, if_fib = =3D 64, if_vnet =3D 0x40000000db, if_home_vnet =3D 0x4000000411, if_vlantrunk =3D 0x40000001bf, if_bpf =3D 0x40000001c1, if_pcount =3D 1051, if_bridge =3D 0x40000001c7, = if_lagg =3D 0x40000003ef, if_pf_kif =3D 0x4000000207, if_carp =3D 0x400000020b, if_label =3D 0x4000= 0002ab, if_netmap =3D 0x4000000215, if_output =3D 0x4000000219, if_input =3D 0x40= 000002af, if_start =3D 0x4000000221, if_ioctl =3D 0x400000022d, if_init =3D 0x40000= 002e1, if_resolvemulti =3D 0x40000002e5, if_qflush =3D 0x4000000305, if_transmit= =3D 0x4000000263, if_reassign =3D 0x4000000265, if_get_counter =3D 0x400000030b, if_request= encap =3D 0x400000026b, if_counters =3D 0xfffff80092e78410, if_hw_tsomax =3D 999, if_hw_tsomaxseg= count =3D 64, if_hw_tsomaxsegsize =3D 735, if_pspare =3D 0xfffff80092e78480, if_hw_addr= =3D 0x4000000039, if_pcp =3D 101 'e', if_bspare =3D 0xfffff80092e784a1 "", if_ispare =3D 0xfffff80092e784a4} (kgdb) frame 11 #11 0xffffffff806723dd in dummynet_send (m=3D0x0) at /home/src/sys/netpfil/ipfw/ip_dn_io.c:774 774 netisr_dispatch(NETISR_IP, m); (kgdb) p m $2 =3D (struct mbuf *) 0x0 (kgdb) l 769 case DIR_OUT: 770 ip_output(m, NULL, NULL, IP_FORWARDING, NUL= L, NULL); 771 break ; 772 773 case DIR_IN : 774 netisr_dispatch(NETISR_IP, m); 775 break; 776 777 #ifdef INET6 778 case DIR_IN | PROTO_IPV6: (kgdb) quit --=20 You are receiving this mail because: You are the assignee for the bug.=