Date: Tue, 08 Dec 2015 16:29:38 -0700 From: Jason Wolfe <j@nitrology.com> To: freebsd-net@freebsd.org Cc: kevin.bowling@kev009.com, hiren@strugglingcoder.info, sbruno@freebsd.org, gnn@freebsd.org Subject: Re: Multiple cores/race conditions in IPv6 RA Message-ID: <cd3a5bda2f93a4af6c49dc40988bad86@m.nitrology.com> In-Reply-To: <50cff74ea38f155ae616cf49f5ffb5ae@m.nitrology.com> References: <50cff74ea38f155ae616cf49f5ffb5ae@m.nitrology.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I believe I've found a way to reproduce this, simply an rtsol against 2 interfaces in isolated loops, which elicits a broadcasted RA. I am able to deadlock the system fairly quickly, which finally results in a core after __rw_wunlock_hard steps in. We have seen this deadlock in one other case, but the code path prior is the same as most of the others, so hopefully it's keeping me on the right track. I tested backing out r243148 without success. load: 51.27 cmd: rtsol 7678 [*Giant] 832.44r 0.00u 0.00s 0% 1868k (prior to core) https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L654 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x30 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff807ada7c stack pointer = 0x28:0xfffffe201edcff60 frame pointer = 0x28:0xfffffe201edcff90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 12 (irq279: ix0:que 1) trap number = 12 panic: page fault cpuid = 1 (kgdb) bt #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff8075fa07 in kern_reboot (howto=16644) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff8075fe05 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758 #3 0xffffffff8075fc93 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:687 #4 0xffffffff80ace3cb in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:851 #5 0xffffffff80ace6cd in trap_pfault (frame=0xfffffe201edcfeb0, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:674 #6 0xffffffff80acdd6a in trap (frame=0xfffffe201edcfeb0) at /usr/src/sys/amd64/amd64/trap.c:440 #7 0xffffffff80ab3d62 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #8 0xffffffff807ada7c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:838 #9 0xffffffff8075e080 in __rw_wunlock_hard (c=0xfffffe1f29f2f3f8, tid=1, file=0x1 <Address 0x1 out of bounds>, line=1) at /usr/src/sys/kern/kern_rwlock.c:988 #10 0xffffffff808a7136 in defrouter_select () at /usr/src/sys/netinet6/nd6_rtr.c:654 #11 0xffffffff808a5a58 in nd6_ra_input (m=<value optimized out>, off=<value optimized out>, icmp6len=<value optimized out>) at /usr/src/sys/netinet6/nd6_rtr.c:804 #12 0xffffffff8087f31f in icmp6_input (mp=<value optimized out>, offp=0xfffffe201edd064c, proto=<value optimized out>) at /usr/src/sys/netinet6/icmp6.c:808 #13 0xffffffff80894adc in ip6_input (m=0xfffff8038cab7000) at /usr/src/sys/netinet6/ip6_input.c:1019 #14 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0x10) at /usr/src/sys/net/netisr.c:976 #15 0xffffffff8082a226 in ether_demux (ifp=<value optimized out>, m=0xfffff8038cab7000) at /usr/src/sys/net/if_ethersubr.c:851 #16 0xffffffff8082aece in ether_nh_input (m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:646 #17 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0x10) at /usr/src/sys/net/netisr.c:976 #18 0xffffffff804c7e89 in ixgbe_rxeof (que=0xfffff8011680d470) at /usr/src/sys/dev/ixgbe/ix_txrx.c:1681 #19 0xffffffff804c1e0a in ixgbe_msix_que (arg=0xfffff8011680d470) at /usr/src/sys/dev/ixgbe/if_ix.c:1391 #20 0xffffffff8072ff9b in intr_event_execute_handlers (p=<value optimized out>, ie=0xfffff80116806600) at /usr/src/sys/kern/kern_intr.c:1264 #21 0xffffffff80730936 in ithread_loop (arg=0xfffff8011681e480) at /usr/src/sys/kern/kern_intr.c:1277 #22 0xffffffff8072dbba in fork_exit (callout=0xffffffff807308a0 <ithread_loop>, arg=0xfffff8011681e480, frame=0xfffffe201edd0ac0) at /usr/src/sys/kern/kern_fork.c:1018 #23 0xffffffff80ab429e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:611 #24 0x0000000000000000 in ?? () (kgdb) frame 10 #10 0xffffffff808a7136 in defrouter_select () at /usr/src/sys/netinet6/nd6_rtr.c:654 654 IF_AFDATA_UNLOCK(dr->ifp); (kgdb) list 649 if (selected_dr == NULL && 650 (ln = nd6_lookup(&dr->rtaddr, 0, dr->ifp)) && 651 ND6_IS_LLINFO_PROBREACH(ln)) { 652 selected_dr = dr; 653 } 654 IF_AFDATA_UNLOCK(dr->ifp); 655 if (ln != NULL) { 656 LLE_RUNLOCK(ln); 657 ln = NULL; 658 } (kgdb) print *dr Cannot access memory at address 0xa00000001 (kgdb) print dr->ifp Cannot access memory at address 0xa00000031 Thanks again, Jason On 2015-12-07 22:32, Jason wrote: > Hi, > > It appears the IPv6 router advertisement code paths were written > fairly lockless, assuming you would never process multiples > concurrently. We are seeing multiple page faults in various places > processing the messages and modifying the routing table. We have > multiple L3 devices and multiple v6 blocks broadcasting these messages > to hardware with dual uplinks in the same VLAN, which I believe is > making us susceptible to this. Though I believe the dual uplink is > all that's required for this, as it can be seen in configurations with > a single v6 block. > > We are running stable/10 @ r285800, and it doesn't appear anything > relevant has changed since then. Our other widely deployed version is > 8.3-RELEASE, which does not see this issue. Upon bumping a machine > from 8.3 -> 10 we can see it start to exhibit this behavior. The only > change I see that might be relevant is r243148, but these cores are > relatively rare, so testing is tough without a considerable > deployment. So basically I'm hoping someone with a trained eye can > send us in the right direction before we go down that road. > > Every backtrace looks pretty much like this, with the location in > nd6_rtr differing: > > panic: page fault > #0 doadump (textdump=1) at pcpu.h:219 > #1 0xffffffff8075fa07 in kern_reboot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:451 > #2 0xffffffff8075fe05 in vpanic (fmt=<value optimized out>, ap=<value > optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758 > #3 0xffffffff8075fc93 in panic (fmt=0x0) at > /usr/src/sys/kern/kern_shutdown.c:687 > #4 0xffffffff80acdf9b in trap_fatal (frame=<value optimized out>, > eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:851 > #5 0xffffffff80ace29d in trap_pfault (frame=0xfffffe0f959b0ff0, > usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:674 > #6 0xffffffff80acd93a in trap (frame=0xfffffe0f959b0ff0) at > /usr/src/sys/amd64/amd64/trap.c:440 > #7 0xffffffff80ab3932 in calltrap () at > /usr/src/sys/amd64/amd64/exception.S:236 > #8 0xffffffff808a5550 in nd6_ra_input (m=<value optimized out>, > off=<value optimized out>, icmp6len=<value optimized out>) > at /usr/src/sys/netinet6/nd6_rtr.c:739 > #9 0xffffffff8087f31f in icmp6_input (mp=<value optimized out>, > offp=0xfffffe0f959b167c, proto=<value optimized out>) > at /usr/src/sys/netinet6/icmp6.c:808 > #10 0xffffffff808949fc in ip6_input (m=0xfffff8002e743200) at > /usr/src/sys/netinet6/ip6_input.c:1019 > #11 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized > out>, source=<value optimized out>, m=0x1) > at /usr/src/sys/net/netisr.c:976 > #12 0xffffffff8082a226 in ether_demux (ifp=<value optimized out>, > m=0xfffff8002e743200) at /usr/src/sys/net/if_ethersubr.c:851 > #13 0xffffffff8082aece in ether_nh_input (m=<value optimized out>) at > /usr/src/sys/net/if_ethersubr.c:646 > #14 0xffffffff80832f02 in netisr_dispatch_src (proto=<value optimized > out>, source=<value optimized out>, m=0x1) > at /usr/src/sys/net/netisr.c:976 > > I'll link to GH for the various relevant bits, because I know everyone > can agree it's the superior RCS. It appears to be that most of these > are caused by the dr struct being freed by concurrent processing: > > https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L578 > https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L654 > https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L728 > https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L739 > https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L800 > https://github.com/freebsd/freebsd/blob/e5ee1c2b414851b17663cb491e2f2317a0af9bda/sys/netinet6/nd6_rtr.c#L1312 > > Thanks for any assistance, > Jason > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?cd3a5bda2f93a4af6c49dc40988bad86>