From owner-freebsd-current@freebsd.org Mon Jul 6 23:02:13 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 320FA995F55 for ; Mon, 6 Jul 2015 23:02:13 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E5431B7B; Mon, 6 Jul 2015 23:02:12 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Type:MIME-Version; bh=WZsnKjnHB7jbKJmCG9TNf9IlJog720K3laloZW8nd0M=; b=Gm5MuGGSCVSaqN+tMSOxgbAMBpSnyKbp6qm/AA69kM75tvm2vyqzciWWbiEbJCY3Ag0vwWY+n8yQ8zXUnFeivY6AyGLoBCxaKmmq0ePkjPbQxHiB95QuNiBo/47k9xTrbx2newMtZWxgFvp9m9lCmYz+fhtfWRu0Ucdee/gHmSE=; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]:42399 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.85 (FreeBSD)) (envelope-from ) id 1ZCFOt-0009Q4-Gr; Mon, 06 Jul 2015 18:02:11 -0500 Received: from 104-54-221-134.lightspeed.austtx.sbcglobal.net ([104.54.221.134]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Mon, 06 Jul 2015 18:02:11 -0500 MIME-Version: 1.0 Date: Mon, 06 Jul 2015 18:02:11 -0500 From: Larry Rosenman To: "K. Macy" Cc: FreeBSD Current , kmacybsd@gmail.com Subject: Re: GPF on Current kernel In-Reply-To: References: <20150705213709.GA47209@borg.lerctr.org> <4eac33a36c1202d20be41e4cbc6b4d75@thebighonker.lerctr.org> Message-ID: <2c94676f5e663925014009c2e66e6533@thebighonker.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.1.1 X-Spam-Score: -1.0 (-) X-LERCTR-Spam-Score: -1.0 (-) X-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 X-LERCTR-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1, SHORTCIRCUIT=-0.0001 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2015 23:02:13 -0000 I've enabled DEBUG_REDZONE, and have DEBUG_MEMGUARD in the system, but have zero clue as to what to set for memguard :( Could I get some guidance? On 2015-07-06 17:43, K. Macy wrote: > That's really disturbing, but could be an artifact of limitations in debug info fidelity. If curthread is corrupt I would expect things to blow up almost instantly. > > Try enabling memguard for the zone or mtype for the object that has the corrupt mutex field. > > -K > On Jul 6, 2015 5:48 AM, "Larry Rosenman" wrote: > actually, the following are interesting further down: > #13 0xffffffff808af473 in sys_write (td=0xdeadc0dedeadc116, > uap=) at /usr/src/sys/kern/sys_generic.c:398 > #14 0xffffffff80b57a6d in amd64_syscall (td=0xfffff80736fda4c0, traced=0) > at subr_syscall.c:133 > > the td value. > > What would you suggest? This has become intermittent :( > > On 2015-07-06 00:42, K. Macy wrote: > The value of the mutex in the stack trace indicates that it's a use after > free. There are various memory debugging options (memguard, redzone) that > may help track it down. > > -K > On Jul 5, 2015 2:37 PM, "Larry Rosenman" wrote: > > I've gotten a couple of these: > > borg.lerctr.org [1] dumped core - see /var/crash/vmcore.3 > > Sun Jul 5 16:32:12 CDT 2015 > > FreeBSD borg.lerctr.org [1] 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r285159: Sun > Jul 5 09:35:33 CDT 2015 root@borg.lerctr.org:/usr/obj/usr/src/sys/LER > amd64 > > panic: general protection fault > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > Fatal trap 9: general protection fault while in kernel mode > cpuid = 0; apic id = 00 > instruction pointer = 0x20:0xffffffff8083a15f > stack pointer = 0x28:0xfffffe100ca1d8a0 > frame pointer = 0x28:0xfffffe100ca1d8e0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 73971 (make) > trap number = 9 > panic: general protection fault > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe100ca1d490 > vpanic() at vpanic+0x189/frame 0xfffffe100ca1d510 > panic() at panic+0x43/frame 0xfffffe100ca1d570 > trap_fatal() at trap_fatal+0x355/frame 0xfffffe100ca1d5d0 > trap() at trap+0x813/frame 0xfffffe100ca1d7e0 > calltrap() at calltrap+0x8/frame 0xfffffe100ca1d7e0 > --- trap 0x9, rip = 0xffffffff8083a15f, rsp = 0xfffffe100ca1d8a0, rbp = > 0xfffffe100ca1d8e0 --- > __mtx_lock_flags() at __mtx_lock_flags+0x4f/frame 0xfffffe100ca1d8e0 > doselwakeup() at doselwakeup+0xb5/frame 0xfffffe100ca1d920 > pipeselwakeup() at pipeselwakeup+0x46/frame 0xfffffe100ca1d940 > pipe_write() at pipe_write+0x13d2/frame 0xfffffe100ca1d9f0 > dofilewrite() at dofilewrite+0x8a/frame 0xfffffe100ca1da40 > kern_writev() at kern_writev+0x68/frame 0xfffffe100ca1da90 > sys_write() at sys_write+0x63/frame 0xfffffe100ca1dae0 > amd64_syscall() at amd64_syscall+0x25d/frame 0xfffffe100ca1dbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100ca1dbf0 > --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x45ae8a, rsp = > 0x7fffffffd328, rbp = 0x7fffffffd350 --- > Uptime: 6h39m51s > Dumping 3186 out of 64460 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/zfs.ko.symbols...done. > Loaded symbols for /boot/kernel/zfs.ko.symbols > Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. > Loaded symbols for /boot/kernel/opensolaris.ko.symbols > Reading symbols from /boot/kernel/linux.ko.symbols...done. > Loaded symbols for /boot/kernel/linux.ko.symbols > Reading symbols from /boot/kernel/linux_common.ko.symbols...done. > Loaded symbols for /boot/kernel/linux_common.ko.symbols > Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. > Loaded symbols for /boot/kernel/if_lagg.ko.symbols > Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. > Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols > Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. > Loaded symbols for /boot/kernel/snd_spicds.ko.symbols > Reading symbols from /boot/kernel/coretemp.ko.symbols...done. > Loaded symbols for /boot/kernel/coretemp.ko.symbols > Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. > Loaded symbols for /boot/kernel/ichsmb.ko.symbols > Reading symbols from /boot/kernel/smbus.ko.symbols...done. > Loaded symbols for /boot/kernel/smbus.ko.symbols > Reading symbols from /boot/kernel/ichwd.ko.symbols...done. > Loaded symbols for /boot/kernel/ichwd.ko.symbols > Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. > Loaded symbols for /boot/kernel/cpuctl.ko.symbols > Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. > Loaded symbols for /boot/kernel/cryptodev.ko.symbols > Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. > Loaded symbols for /boot/kernel/dtraceall.ko.symbols > Reading symbols from /boot/kernel/profile.ko.symbols...done. > Loaded symbols for /boot/kernel/profile.ko.symbols > Reading symbols from /boot/kernel/dtrace.ko.symbols...done. > Loaded symbols for /boot/kernel/dtrace.ko.symbols > Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. > Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols > Reading symbols from /boot/kernel/systrace.ko.symbols...done. > Loaded symbols for /boot/kernel/systrace.ko.symbols > Reading symbols from /boot/kernel/sdt.ko.symbols...done. > Loaded symbols for /boot/kernel/sdt.ko.symbols > Reading symbols from /boot/kernel/lockstat.ko.symbols...done. > Loaded symbols for /boot/kernel/lockstat.ko.symbols > Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. > Loaded symbols for /boot/kernel/fasttrap.ko.symbols > Reading symbols from /boot/kernel/fbt.ko.symbols...done. > Loaded symbols for /boot/kernel/fbt.ko.symbols > Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. > Loaded symbols for /boot/kernel/dtnfscl.ko.symbols > Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. > Loaded symbols for /boot/kernel/dtmalloc.ko.symbols > Reading symbols from /boot/modules/vboxdrv.ko...done. > Loaded symbols for /boot/modules/vboxdrv.ko > Reading symbols from /boot/modules/nvidia.ko...done. > Loaded symbols for /boot/modules/nvidia.ko > Reading symbols from /boot/kernel/ipmi.ko.symbols...done. > Loaded symbols for /boot/kernel/ipmi.ko.symbols > Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. > Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols > Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. > Loaded symbols for /boot/kernel/radeonkms.ko.symbols > Reading symbols from /boot/kernel/iicbb.ko.symbols...done. > Loaded symbols for /boot/kernel/iicbb.ko.symbols > Reading symbols from /boot/kernel/iicbus.ko.symbols...done. > Loaded symbols for /boot/kernel/iicbus.ko.symbols > Reading symbols from /boot/kernel/iic.ko.symbols...done. > Loaded symbols for /boot/kernel/iic.ko.symbols > Reading symbols from /boot/kernel/drm2.ko.symbols...done. > Loaded symbols for /boot/kernel/drm2.ko.symbols > Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. > Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols > Reading symbols from /boot/kernel/uhid.ko.symbols...done. > Loaded symbols for /boot/kernel/uhid.ko.symbols > Reading symbols from /boot/modules/vboxnetflt.ko...done. > Loaded symbols for /boot/modules/vboxnetflt.ko > Reading symbols from /boot/kernel/netgraph.ko.symbols...done. > Loaded symbols for /boot/kernel/netgraph.ko.symbols > Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. > Loaded symbols for /boot/kernel/ng_ether.ko.symbols > Reading symbols from /boot/modules/vboxnetadp.ko...done. > Loaded symbols for /boot/modules/vboxnetadp.ko > #0 doadump (textdump=1) at pcpu.h:221 > 221 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=1) at pcpu.h:221 > #1 0xffffffff80855d55 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:447 > #2 0xffffffff80856348 in vpanic (fmt=, > ap=) at /usr/src/sys/kern/kern_shutdown.c:744 > #3 0xffffffff80856393 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:675 > #4 0xffffffff80b57275 in trap_fatal (frame=, > eva=) at /usr/src/sys/amd64/amd64/trap.c:853 > #5 0xffffffff80b56f03 in trap (frame=) > at /usr/src/sys/amd64/amd64/trap.c:201 > #6 0xffffffff80b37142 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:235 > #7 0xffffffff8083a15f in __mtx_lock_flags (c=0xdeadc0dedeadc116, opts=0, > file=0xffffffff80da2bc8 "/usr/src/sys/kern/sys_generic.c", line=1808) > at /usr/src/sys/kern/kern_mutex.c:211 > #8 0xffffffff808b1b25 in doselwakeup (sip=0xfffff8019bd5b0b8, pri=104) > at /usr/src/sys/kern/sys_generic.c:1808 > #9 0xffffffff808b4b86 in pipeselwakeup (cpipe=0xfffff8019bd5b000) > at /usr/src/sys/kern/sys_pipe.c:622 > #10 0xffffffff808b3892 in pipe_write (fp=0xfffff80027c3d280, > uio=, active_cred=, > flags=1808, > td=0xfffff80736fda4c0) at /usr/src/sys/kern/sys_pipe.c:1314 > #11 0xffffffff808af7da in dofilewrite (td=0xfffff80736fda4c0, fd=1, > fp=0xfffff80027c3d280, auio=0xfffffe100ca1dab0, > offset=, flags=0) at file.h:310 > #12 0xffffffff808af4e8 in kern_writev (td=0xfffff80736fda4c0, fd=1, > auio=0xfffffe100ca1dab0) at /usr/src/sys/kern/sys_generic.c:485 > #13 0xffffffff808af473 in sys_write (td=0xdeadc0dedeadc116, > uap=) at /usr/src/sys/kern/sys_generic.c:398 > #14 0xffffffff80b57a6d in amd64_syscall (td=0xfffff80736fda4c0, traced=0) > at subr_syscall.c:133 > #15 0xffffffff80b3742b in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:395 > #16 0x000000000045ae8a in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) > > Ideas? > > FreeBSD borg.lerctr.org [1] 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r285159: Sun > Jul 5 09:35:33 CDT 2015 root@borg.lerctr.org:/usr/obj/usr/src/sys/LER > amd64 1100077 1100077 > -- > Larry Rosenman http://www.lerctr.org/~ler [2] > Phone: +1 214-642-9640 [3] E-Mail: ler@lerctr.org > US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current [4] > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current [4] > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- Larry Rosenman http://www.lerctr.org/~ler [2] Phone: +1 214-642-9640 [3] E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 -- Larry Rosenman http://www.lerctr.org/~ler [2] Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 Links: ------ [1] http://borg.lerctr.org [2] http://www.lerctr.org/~ler [3] tel:%2B1%20214-642-9640 [4] http://lists.freebsd.org/mailman/listinfo/freebsd-current