From owner-freebsd-net@freebsd.org Wed Aug 15 08:49:04 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A16DF107A2DE for ; Wed, 15 Aug 2018 08:49:04 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 568D880BB1; Wed, 15 Aug 2018 08:49:04 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 094A812CBF; Wed, 15 Aug 2018 08:49:04 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [192.168.14.247] (unknown [62.49.66.12]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id B7AED591EA; Wed, 15 Aug 2018 10:49:01 +0200 (CEST) From: "Kristof Provost" To: "Matthew Macy" Cc: freebsd-net@freebsd.org Subject: Re: Panic during ci test run Date: Wed, 15 Aug 2018 09:49:00 +0100 X-Mailer: MailMate (2.0BETAr6116) Message-ID: <81370B9F-12D6-4270-9569-720EB49351CF@FreeBSD.org> In-Reply-To: References: <34C6043C-FDD2-4812-AFF3-C61DEF7AE435@FreeBSD.org> MIME-Version: 1.0 Embedded-HTML: [{"HTML":[711, 6600], "plain":[301, 5614], "uuid":"1F1996E5-2272-4182-94F1-3193A4566BA4"}] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Aug 2018 08:49:04 -0000 That’s odd. With the appropriate flags set the panic is completely reliable for me. Does the test just succeed for you, or does it skip? I don’t have any local changes, and it reproduces on the ci.freebsd.org tests in any case. Regards, Kristof On 14 Aug 2018, at 23:42, Matthew Macy wrote: > This isn't reproducing it for me. I'll need more specifics on your > configuration. > -M > > On Sat, Aug 11, 2018 at 2:04 AM Kristof Provost > wrote: > >> The fibs_test:subnet_route_with_multiple_fibs_on_same_subnet test >> (/usr/tests/sys/netinet/) consistently provokes a panic. >> >> Note that this requires: >> >> - test_suites.FreeBSD.fibs = '1 2' in >> /usr/local/etc/kyua/kyua.conf >> - net.fibs=3 in /boot/loader.conf >> - sysctl net.add_addr_allfibs=0 >> >> Then: >> >> - cd /usr/tests/sys/netinet/ >> - sudo kyua test >> >> This results in: >> >> Fatal trap 9: general protection fault while in kernel mode >> cpuid = 2; apic id = 02 >> instruction pointer = 0x20:0xffffffff80ded4c3 >> stack pointer = 0x28:0xfffffe0000427860 >> frame pointer = 0x28:0xfffffe00004278a0 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 0 (softirq_2) >> [ thread pid 0 tid 100021 ] >> Stopped at inp_gcmoptions+0xe3: movq ll+0x33f(%rax),%r9 >> db> bt >> Tracing pid 0 tid 100021 td 0xfffff80004605000 >> inp_gcmoptions() at inp_gcmoptions+0xe3/frame 0xfffffe00004278a0 >> epoch_call_task() at epoch_call_task+0x21a/frame 0xfffffe00004278f0 >> gtaskqueue_run_locked() at gtaskqueue_run_locked+0x139/frame >> 0xfffffe0000427940 >> gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x88/frame >> 0xfffffe0000427970 >> fork_exit() at fork_exit+0x84/frame 0xfffffe00004279b0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004279b0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> >> kgdb decodes that to: >> >> #0 __curthread () at ./machine/pcpu.h:230 >> #1 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:366 >> #2 0xffffffff8043dd4b in db_dump (dummy=, >> dummy2=, dummy3=, dummy4=) at >> /usr/src/sys/ddb/db_command.c:574 >> #3 0xffffffff8043db19 in db_command (last_cmdp=, >> cmd_table=, dopager=) at >> /usr/src/sys/ddb/db_command.c:481 >> #4 0xffffffff8043d894 in db_command_loop () at >> /usr/src/sys/ddb/db_command.c:534 >> #5 0xffffffff80440abf in db_trap (type=, >> code=) at /usr/src/sys/ddb/db_main.c:252 >> #6 0xffffffff80bdef43 in kdb_trap (type=9, code=0, tf=> out>) at /usr/src/sys/kern/subr_kdb.c:693 >> #7 0xffffffff8107aee1 in trap_fatal (frame=0xfffffe00004277a0, >> eva=0) at /usr/src/sys/amd64/amd64/trap.c:906 >> #8 0xffffffff8107a3bd in trap (frame=0xfffffe00004277a0) at >> /usr/src/sys/amd64/amd64/trap.c:203 >> #9 >> #10 inp_gcmoptions (ctx=0xfffff800142da5e0) at >> /usr/src/sys/netinet6/in6_mcast.c:1650 >> #11 0xffffffff80bd9c7a in epoch_call_task (arg=) at >> /usr/src/sys/kern/subr_epoch.c:507 >> #12 0xffffffff80bdd069 in gtaskqueue_run_locked >> (queue=0xfffff800040ceb00) at /usr/src/sys/kern/subr_gtaskqueue.c:332 >> #13 0xffffffff80bdcde8 in gtaskqueue_thread_loop (arg=> out>) at /usr/src/sys/kern/subr_gtaskqueue.c:507 >> #14 0xffffffff80b53084 in fork_exit (callout=0xffffffff80bdcd60 >> , arg=0xfffffe0087e40038, >> frame=0xfffffe00004279c0) at /usr/src/sys/kern/kern_fork.c:1057 >> #15 >> >> It looks like the inm has been freed at that point, so we try to >> dereference a freed pointer, and that doesn’t go well for us: >> >> (kgdb) fr 10 >> #10 inp_gcmoptions (ctx=0xfffff800142da5e0) at >> /usr/src/sys/netinet6/in6_mcast.c:1650 >> 1650 CURVNET_SET(ifp->if_vnet); >> (kgdb) p ifp >> $1 = (struct ifnet *) 0xdeadc0dedeadc0de >> (kgdb) >> (kgdb) l >> 1645 if (imf) >> 1646 im6f_leave(imf); >> 1647 inm = imo->im6o_membership[idx]; >> 1648 ifp = inm->in6m_ifp; >> 1649 if (ifp != NULL) { >> 1650 CURVNET_SET(ifp->if_vnet); >> 1651 (void)in6_leavegroup(inm, imf); >> 1652 CURVNET_RESTORE(); >> 1653 } else { >> 1654 (void)in6_leavegroup(inm, imf); >> (kgdb) p inm >> $2 = (struct in6_multi *) 0xfffff8001435b200 >> (kgdb) p *inm >> $3 = {in6m_addr = {__u6_addr = {__u6_addr8 = >> "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255", >> , __u6_addr16 = {49374, 57005, 49374, >> 57005, 49374, 57005, 49374, 57005}, __u6_addr32 = >> {3735929054, 3735929054, 3735929054, 3735929054}}}, in6m_ifp = >> 0xdeadc0dedeadc0de, in6m_ifma = 0xdeadc0dedeadc0de, >> in6m_refcount = 3735929054, in6m_state = 3735929054, in6m_timer = >> 3735929054, in6m_mli = 0xdeadc0dedeadc0de, in6m_nrele = {sle_next = >> 0xdeadc0dedeadc0de}, in6m_srcs = { >> rbh_root = 0xdeadc0dedeadc0de}, in6m_nsrc = 16045693110842147038, >> in6m_scq = {mq_head = {stqh_first = 0xdeadc0dedeadc0de, stqh_last = >> 0xdeadc0dedeadc0de}, >> mq_len = -559038242, mq_maxlen = -559038242}, in6m_lastgsrtv = >> {tv_sec = -2401050962867404578, tv_usec = -2401050962867404578}, >> in6m_sctimer = 49374, in6m_scrv = 57005, >> in6m_st = {{iss_fmode = 49374, iss_asm = 57005, iss_ex = 49374, >> iss_in = 57005, iss_rec = 49374}, {iss_fmode = 57005, iss_asm = >> 49374, iss_ex = 57005, iss_in = 49374, >> iss_rec = 57005}}} >> (kgdb) >> (kgdb) p nmships >> $4 = 1 >> (kgdb) p *imf >> $6 = {im6f_sources = {rbh_root = 0x0}, im6f_nsrc = 0, im6f_st = >> "\002\001"} >> (kgdb) >> >> Regards, >> Kristof >>