From owner-freebsd-net@freebsd.org Tue Aug 14 22:42:23 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F55A106861A for ; Tue, 14 Aug 2018 22:42:23 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C65258C6F5; Tue, 14 Aug 2018 22:42:22 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 87FEEEFAB; Tue, 14 Aug 2018 22:42:22 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it0-f54.google.com with SMTP id g141-v6so22444225ita.4; Tue, 14 Aug 2018 15:42:22 -0700 (PDT) X-Gm-Message-State: AOUpUlEtCeIZ3TcpdoulALvwydg1w7aC+gBTHWvDtma8zgfBJM/c+iat 1ejH8/qdIiDJCYjPTRK/g2JBbVvhmwaXUcdTblk= X-Google-Smtp-Source: AA+uWPzmxz8KzLS6SzaH7N0C60I/Y91Vig4quXD0hQdqGh1qorNBI/LNtluRTb78wMi7ESkuweeduJwatq3JY9dJdBg= X-Received: by 2002:a02:2b12:: with SMTP id h18-v6mr20963321jaa.10.1534286541699; Tue, 14 Aug 2018 15:42:21 -0700 (PDT) MIME-Version: 1.0 References: <34C6043C-FDD2-4812-AFF3-C61DEF7AE435@FreeBSD.org> In-Reply-To: <34C6043C-FDD2-4812-AFF3-C61DEF7AE435@FreeBSD.org> From: Matthew Macy Date: Tue, 14 Aug 2018 15:42:10 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Panic during ci test run To: kp@freebsd.org Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Aug 2018 22:42:23 -0000 This isn't reproducing it for me. I'll need more specifics on your configuration. -M On Sat, Aug 11, 2018 at 2:04 AM Kristof Provost wrote: > The fibs_test:subnet_route_with_multiple_fibs_on_same_subnet test > (/usr/tests/sys/netinet/) consistently provokes a panic. > > Note that this requires: > > - test_suites.FreeBSD.fibs =3D '1 2' in /usr/local/etc/kyua/kyua.conf > - net.fibs=3D3 in /boot/loader.conf > - sysctl net.add_addr_allfibs=3D0 > > Then: > > - cd /usr/tests/sys/netinet/ > - sudo kyua test > > This results in: > > Fatal trap 9: general protection fault while in kernel mode > cpuid =3D 2; apic id =3D 02 > instruction pointer =3D 0x20:0xffffffff80ded4c3 > stack pointer =3D 0x28:0xfffffe0000427860 > frame pointer =3D 0x28:0xfffffe00004278a0 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 0 (softirq_2) > [ thread pid 0 tid 100021 ] > Stopped at inp_gcmoptions+0xe3: movq ll+0x33f(%rax),%r9 > db> bt > Tracing pid 0 tid 100021 td 0xfffff80004605000 > inp_gcmoptions() at inp_gcmoptions+0xe3/frame 0xfffffe00004278a0 > epoch_call_task() at epoch_call_task+0x21a/frame 0xfffffe00004278f0 > gtaskqueue_run_locked() at gtaskqueue_run_locked+0x139/frame 0xfffffe0000= 427940 > gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x88/frame 0xfffffe000= 0427970 > fork_exit() at fork_exit+0x84/frame 0xfffffe00004279b0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004279b0 > --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- > > kgdb decodes that to: > > #0 __curthread () at ./machine/pcpu.h:230 > #1 doadump (textdump=3D0) at /usr/src/sys/kern/kern_shutdown.c:366 > #2 0xffffffff8043dd4b in db_dump (dummy=3D, dummy2=3D, dummy3=3D, dummy4=3D) at /usr/src/sys/= ddb/db_command.c:574 > #3 0xffffffff8043db19 in db_command (last_cmdp=3D, cmd_ta= ble=3D, dopager=3D) at /usr/src/sys/ddb/db_co= mmand.c:481 > #4 0xffffffff8043d894 in db_command_loop () at /usr/src/sys/ddb/db_comma= nd.c:534 > #5 0xffffffff80440abf in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:252 > #6 0xffffffff80bdef43 in kdb_trap (type=3D9, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:693 > #7 0xffffffff8107aee1 in trap_fatal (frame=3D0xfffffe00004277a0, eva=3D0= ) at /usr/src/sys/amd64/amd64/trap.c:906 > #8 0xffffffff8107a3bd in trap (frame=3D0xfffffe00004277a0) at /usr/src/s= ys/amd64/amd64/trap.c:203 > #9 > #10 inp_gcmoptions (ctx=3D0xfffff800142da5e0) at /usr/src/sys/netinet6/in= 6_mcast.c:1650 > #11 0xffffffff80bd9c7a in epoch_call_task (arg=3D) at /usr= /src/sys/kern/subr_epoch.c:507 > #12 0xffffffff80bdd069 in gtaskqueue_run_locked (queue=3D0xfffff800040ceb= 00) at /usr/src/sys/kern/subr_gtaskqueue.c:332 > #13 0xffffffff80bdcde8 in gtaskqueue_thread_loop (arg=3D) = at /usr/src/sys/kern/subr_gtaskqueue.c:507 > #14 0xffffffff80b53084 in fork_exit (callout=3D0xffffffff80bdcd60 , arg=3D0xfffffe0087e40038, frame=3D0xfffffe00004279c0) at= /usr/src/sys/kern/kern_fork.c:1057 > #15 > > It looks like the inm has been freed at that point, so we try to > dereference a freed pointer, and that doesn=E2=80=99t go well for us: > > (kgdb) fr 10 > #10 inp_gcmoptions (ctx=3D0xfffff800142da5e0) at /usr/src/sys/netinet6/in= 6_mcast.c:1650 > 1650 CURVNET_SET(ifp->if_vnet); > (kgdb) p ifp > $1 =3D (struct ifnet *) 0xdeadc0dedeadc0de > (kgdb) > (kgdb) l > 1645 if (imf) > 1646 im6f_leave(imf); > 1647 inm =3D imo->im6o_membership[idx]; > 1648 ifp =3D inm->in6m_ifp; > 1649 if (ifp !=3D NULL) { > 1650 CURVNET_SET(ifp->if_vnet); > 1651 (void)in6_leavegroup(inm, imf); > 1652 CURVNET_RESTORE(); > 1653 } else { > 1654 (void)in6_leavegroup(inm, imf); > (kgdb) p inm > $2 =3D (struct in6_multi *) 0xfffff8001435b200 > (kgdb) p *inm > $3 =3D {in6m_addr =3D {__u6_addr =3D {__u6_addr8 =3D "\336\300\255\336\33= 6\300\255\336\336\300\255\336\336\300\255", , __u= 6_addr16 =3D {49374, 57005, 49374, > 57005, 49374, 57005, 49374, 57005}, __u6_addr32 =3D {3735929054, = 3735929054, 3735929054, 3735929054}}}, in6m_ifp =3D 0xdeadc0dedeadc0de, in6= m_ifma =3D 0xdeadc0dedeadc0de, > in6m_refcount =3D 3735929054, in6m_state =3D 3735929054, in6m_timer =3D= 3735929054, in6m_mli =3D 0xdeadc0dedeadc0de, in6m_nrele =3D {sle_next =3D = 0xdeadc0dedeadc0de}, in6m_srcs =3D { > rbh_root =3D 0xdeadc0dedeadc0de}, in6m_nsrc =3D 16045693110842147038,= in6m_scq =3D {mq_head =3D {stqh_first =3D 0xdeadc0dedeadc0de, stqh_last = =3D 0xdeadc0dedeadc0de}, > mq_len =3D -559038242, mq_maxlen =3D -559038242}, in6m_lastgsrtv =3D = {tv_sec =3D -2401050962867404578, tv_usec =3D -2401050962867404578}, in6m_s= ctimer =3D 49374, in6m_scrv =3D 57005, > in6m_st =3D {{iss_fmode =3D 49374, iss_asm =3D 57005, iss_ex =3D 49374,= iss_in =3D 57005, iss_rec =3D 49374}, {iss_fmode =3D 57005, iss_asm =3D 49= 374, iss_ex =3D 57005, iss_in =3D 49374, > iss_rec =3D 57005}}} > (kgdb) > (kgdb) p nmships > $4 =3D 1 > (kgdb) p *imf > $6 =3D {im6f_sources =3D {rbh_root =3D 0x0}, im6f_nsrc =3D 0, im6f_st =3D= "\002\001"} > (kgdb) > > Regards, > Kristof >