From owner-freebsd-net@freebsd.org Sat Aug 11 09:04:40 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2E521063C16 for ; Sat, 11 Aug 2018 09:04:40 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 610BE727F7; Sat, 11 Aug 2018 09:04:40 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 130BEE6B8; Sat, 11 Aug 2018 09:04:40 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [172.28.128.1] (p5B34BA34.dip0.t-ipconnect.de [91.52.186.52]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 684F44C4C6; Sat, 11 Aug 2018 11:04:38 +0200 (CEST) From: "Kristof Provost" To: "FreeBSD Net" Cc: mmacy@FreeBSD.org Subject: Panic during ci test run Date: Sat, 11 Aug 2018 11:04:37 +0200 X-Mailer: MailMate (2.0BETAr6116) Message-ID: <34C6043C-FDD2-4812-AFF3-C61DEF7AE435@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Aug 2018 09:04:41 -0000 The fibs_test:subnet_route_with_multiple_fibs_on_same_subnet test (/usr/tests/sys/netinet/) consistently provokes a panic. Note that this requires: - test_suites.FreeBSD.fibs = '1 2' in /usr/local/etc/kyua/kyua.conf - net.fibs=3 in /boot/loader.conf - sysctl net.add_addr_allfibs=0 Then: - cd /usr/tests/sys/netinet/ - sudo kyua test This results in: Fatal trap 9: general protection fault while in kernel mode cpuid = 2; apic id = 02 instruction pointer = 0x20:0xffffffff80ded4c3 stack pointer = 0x28:0xfffffe0000427860 frame pointer = 0x28:0xfffffe00004278a0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (softirq_2) [ thread pid 0 tid 100021 ] Stopped at inp_gcmoptions+0xe3: movq ll+0x33f(%rax),%r9 db> bt Tracing pid 0 tid 100021 td 0xfffff80004605000 inp_gcmoptions() at inp_gcmoptions+0xe3/frame 0xfffffe00004278a0 epoch_call_task() at epoch_call_task+0x21a/frame 0xfffffe00004278f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x139/frame 0xfffffe0000427940 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x88/frame 0xfffffe0000427970 fork_exit() at fork_exit+0x84/frame 0xfffffe00004279b0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004279b0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- kgdb decodes that to: #0 __curthread () at ./machine/pcpu.h:230 #1 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff8043dd4b in db_dump (dummy=, dummy2=, dummy3=, dummy4=) at /usr/src/sys/ddb/db_command.c:574 #3 0xffffffff8043db19 in db_command (last_cmdp=, cmd_table=, dopager=) at /usr/src/sys/ddb/db_command.c:481 #4 0xffffffff8043d894 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534 #5 0xffffffff80440abf in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:252 #6 0xffffffff80bdef43 in kdb_trap (type=9, code=0, tf=) at /usr/src/sys/kern/subr_kdb.c:693 #7 0xffffffff8107aee1 in trap_fatal (frame=0xfffffe00004277a0, eva=0) at /usr/src/sys/amd64/amd64/trap.c:906 #8 0xffffffff8107a3bd in trap (frame=0xfffffe00004277a0) at /usr/src/sys/amd64/amd64/trap.c:203 #9 #10 inp_gcmoptions (ctx=0xfffff800142da5e0) at /usr/src/sys/netinet6/in6_mcast.c:1650 #11 0xffffffff80bd9c7a in epoch_call_task (arg=) at /usr/src/sys/kern/subr_epoch.c:507 #12 0xffffffff80bdd069 in gtaskqueue_run_locked (queue=0xfffff800040ceb00) at /usr/src/sys/kern/subr_gtaskqueue.c:332 #13 0xffffffff80bdcde8 in gtaskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_gtaskqueue.c:507 #14 0xffffffff80b53084 in fork_exit (callout=0xffffffff80bdcd60 , arg=0xfffffe0087e40038, frame=0xfffffe00004279c0) at /usr/src/sys/kern/kern_fork.c:1057 #15 It looks like the inm has been freed at that point, so we try to dereference a freed pointer, and that doesn’t go well for us: (kgdb) fr 10 #10 inp_gcmoptions (ctx=0xfffff800142da5e0) at /usr/src/sys/netinet6/in6_mcast.c:1650 1650 CURVNET_SET(ifp->if_vnet); (kgdb) p ifp $1 = (struct ifnet *) 0xdeadc0dedeadc0de (kgdb) (kgdb) l 1645 if (imf) 1646 im6f_leave(imf); 1647 inm = imo->im6o_membership[idx]; 1648 ifp = inm->in6m_ifp; 1649 if (ifp != NULL) { 1650 CURVNET_SET(ifp->if_vnet); 1651 (void)in6_leavegroup(inm, imf); 1652 CURVNET_RESTORE(); 1653 } else { 1654 (void)in6_leavegroup(inm, imf); (kgdb) p inm $2 = (struct in6_multi *) 0xfffff8001435b200 (kgdb) p *inm $3 = {in6m_addr = {__u6_addr = {__u6_addr8 = "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255", , __u6_addr16 = {49374, 57005, 49374, 57005, 49374, 57005, 49374, 57005}, __u6_addr32 = {3735929054, 3735929054, 3735929054, 3735929054}}}, in6m_ifp = 0xdeadc0dedeadc0de, in6m_ifma = 0xdeadc0dedeadc0de, in6m_refcount = 3735929054, in6m_state = 3735929054, in6m_timer = 3735929054, in6m_mli = 0xdeadc0dedeadc0de, in6m_nrele = {sle_next = 0xdeadc0dedeadc0de}, in6m_srcs = { rbh_root = 0xdeadc0dedeadc0de}, in6m_nsrc = 16045693110842147038, in6m_scq = {mq_head = {stqh_first = 0xdeadc0dedeadc0de, stqh_last = 0xdeadc0dedeadc0de}, mq_len = -559038242, mq_maxlen = -559038242}, in6m_lastgsrtv = {tv_sec = -2401050962867404578, tv_usec = -2401050962867404578}, in6m_sctimer = 49374, in6m_scrv = 57005, in6m_st = {{iss_fmode = 49374, iss_asm = 57005, iss_ex = 49374, iss_in = 57005, iss_rec = 49374}, {iss_fmode = 57005, iss_asm = 49374, iss_ex = 57005, iss_in = 49374, iss_rec = 57005}}} (kgdb) (kgdb) p nmships $4 = 1 (kgdb) p *imf $6 = {im6f_sources = {rbh_root = 0x0}, im6f_nsrc = 0, im6f_st = "\002\001"} (kgdb) Regards, Kristof