From owner-freebsd-stable@freebsd.org Mon Oct 12 12:32:47 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20EE9A119BD for ; Mon, 12 Oct 2015 12:32:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 8B00A15F5; Mon, 12 Oct 2015 12:32:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:1DnlFRYzMzvfcNtWwb1nMBP/LSx+4OfEezUN459isYplN5qZpcu5bnLW6fgltlLVR4KTs6sC0LqK9fC6EjBQqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh7z0q8eYP1UArQH+SI0xBS3+lR/WuMgSjNkqAYcK4TyNnEF1ff9Lz3hjP1OZkkW0zM6x+Jl+73YY4Kp5pIZoGJ/3dKUgTLFeEC9ucyVsvJWq5lH/Sl6k5mEdSXkXmxwAIBLM8AP3RN+luSjSvelm3yeGe8H7G+MaQzOnup1qQxygrS4MNDo09SmDkMl5h6FfrReJuhtw3oPQeIHTP/MoLfCVRs8TWWcUBpUZbCdGGI7pKtJXV+c= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CTAgBNpxtW/61jaINVCIN6bga9fQENgVoXCoJyggo1SgKBahQBAQEBAQEBAYEJgh+CBwEBAQMBAQEBIAQnIAsFCQICAQgOCgICDRkCAhsMAQkmAgQIBwQBHASIBQgNqgqTTQEBAQEBAQEDAQEBAQEBAQEBARgEgR6FUYR+hCoMBAIBHDQHgmmBRQWHOYcFhxVAhRmFGYRASINykgKDbQIfAQFChB4iMweGXYEGAQEB X-IronPort-AV: E=Sophos;i="5.17,672,1437451200"; d="scan'208";a="244026934" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 12 Oct 2015 08:32:38 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7EA0215F55D; Mon, 12 Oct 2015 08:32:38 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id qH2iENJUPRve; Mon, 12 Oct 2015 08:32:37 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 42CA115F565; Mon, 12 Oct 2015 08:32:37 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XhpcumsKmHs5; Mon, 12 Oct 2015 08:32:37 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 2300F15F55D; Mon, 12 Oct 2015 08:32:37 -0400 (EDT) Date: Mon, 12 Oct 2015 08:32:36 -0400 (EDT) From: Rick Macklem To: Christian Kratzer Cc: freebsd-stable@freebsd.org, John Baldwin Message-ID: <2135054744.32546564.1444653156980.JavaMail.zimbra@uoguelph.ca> In-Reply-To: References: <2148690.gx9M0ZzrG1@ralph.baldwin.cx> <3563189.eDHDcCgW5L@ralph.baldwin.cx> <358885214.31305796.1444518367048.JavaMail.zimbra@uoguelph.ca> Subject: Re: smbfs crashes since approx. 10.1-RELEASE MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: smbfs crashes since approx. 10.1-RELEASE Thread-Index: zbjHYEpvbXNqKVqi1WrwcfoRNLghjA== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 12:32:47 -0000 Christian Kratzer wrote: > Hi Rick, > > there was also a second more recent crash in /var/crash > > Mon Oct 12 03:01:16 CEST 2015 > > FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M: Sun > Oct 11 08:37:40 CEST 2015 > ck@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC amd64 > > panic: Assertion mtx_unowned(m) failed at > /usr/src/sys/kern/kern_mutex.c:955 > Oops, I screwed up. I should have looked at this panic assertion when you reported it before. Ok, so if I understand the assertion correctly, it means that another thread has the mutex locked. If this is correct, I'll have to take another look at the code and figure out how to wait for these other threads to finish with the mutexes. I do think the patch fixes the race I saw, but there must be other races in the code. I'll take another look, but if anyone else is conversant with netsmb, feel free to jump in, because it is all new to me. Unfortunately, I won't have any way to do testing for the next month or so, so any patches I do come up with will be "try this untested..". rick > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > panic: Assertion mtx_unowned(m) failed at > /usr/src/sys/kern/kern_mutex.c:955 > cpuid = 3 > KDB: stack backtrace: > #0 0xffffffff80975bb0 at kdb_backtrace+0x60 > #1 0xffffffff8093baa6 at vpanic+0x126 > #2 0xffffffff8093b979 at kassert_panic+0x139 > #3 0xffffffff80921c47 at _mtx_destroy+0x77 > #4 0xffffffff81a1c174 at smb_iod_destroy+0xc4 > #5 0xffffffff81a12eea at smb_vc_free+0x1a > #6 0xffffffff81a13e24 at sdp_trydestroy+0xb4 > #7 0xffffffff81a1cb96 at smbfs_unmount+0xd6 > #8 0xffffffff809d9e84 at dounmount+0x524 > #9 0xffffffff809d9936 at sys_unmount+0x3c6 > #10 0xffffffff80d42235 at amd64_syscall+0x265 > #11 0xffffffff80d25cfb at Xfast_syscall+0xfb > Uptime: 8h44m10s > Dumping 102 out of 999 MB:..16%..32%..47%..63%..78%..94% > > Reading symbols from /boot/kernel/smbfs.ko.symbols...done. > Loaded symbols for /boot/kernel/smbfs.ko.symbols > Reading symbols from /boot/kernel/libiconv.ko.symbols...done. > Loaded symbols for /boot/kernel/libiconv.ko.symbols > Reading symbols from /boot/kernel/libmchain.ko.symbols...done. > Loaded symbols for /boot/kernel/libmchain.ko.symbols > #0 doadump (textdump=) at pcpu.h:219 > 219 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=) at pcpu.h:219 > #1 0xffffffff8093b5f2 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:451 > #2 0xffffffff8093bae5 in vpanic (fmt=, > ap=) at /usr/src/sys/kern/kern_shutdown.c:758 > #3 0xffffffff8093b979 in kassert_panic (fmt=) > at /usr/src/sys/kern/kern_shutdown.c:646 > #4 0xffffffff80921c47 in _mtx_destroy (c=0xfffff80002db5490) > at /usr/src/sys/kern/kern_mutex.c:955 > #5 0xffffffff81a1c174 in smb_iod_destroy (iod=0xfffff80002db5400) > at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733 > #6 0xffffffff81a12eea in smb_vc_free (cp=0xfffff80002933000) > at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499 > #7 0xffffffff81a13e24 in sdp_trydestroy (sdp=0xfffff80002904140) > at /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166 > #8 0xffffffff81a1cb96 in smbfs_unmount (mp=0xfffff80015226000, > mntflags=) > at /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297 > #9 0xffffffff809d9e84 in dounmount (mp=0xfffff80015226000, > flags=134217728, > td=0xfffff800151b0940) at /usr/src/sys/kern/vfs_mount.c:1313 > #10 0xffffffff809d9936 in sys_unmount (td=0xfffff800151b0940, > uap=0xfffffe003d643b80) at /usr/src/sys/kern/vfs_mount.c:1205 > #11 0xffffffff80d42235 in amd64_syscall (td=0xfffff800151b0940, > traced=0) > at subr_syscall.c:134 > #12 0xffffffff80d25cfb in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:396 > #13 0x000000080089190a in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) > > > Greetings > Christian > > > > > > > > > > > On Mon, 12 Oct 2015, Christian Kratzer wrote: > > > Hi Rick, > > > > On Sat, 10 Oct 2015, Rick Macklem wrote: > > > >> Hi again, > >> > >> Attached is a semantically equivalent patch to the one I posted a few > >> minutes ago, but I think this one is more readable. > >> > >> Please let me know if you get it tested, rick > > > > the box crashed again tonight with your patch applied. Here's the new > > crashinfo: > > > > panic: Assertion mtx_unowned(m) failed at > > /usr/src/sys/kern/kern_mutex.c:955 > > > > GNU gdb 6.1.1 [FreeBSD] > > Copyright 2004 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, and you > > are > > welcome to change it and/or distribute copies of it under certain > > conditions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for > > details. > > This GDB was configured as "amd64-marcel-freebsd"... > > > > Unread portion of the kernel message buffer: > > panic: Assertion mtx_unowned(m) failed at > > /usr/src/sys/kern/kern_mutex.c:955 > > cpuid = 2 > > KDB: stack backtrace: > > #0 0xffffffff80975bb0 at kdb_backtrace+0x60 > > #1 0xffffffff8093baa6 at vpanic+0x126 > > #2 0xffffffff8093b979 at kassert_panic+0x139 > > #3 0xffffffff80921c47 at _mtx_destroy+0x77 > > #4 0xffffffff81a1c114 at smb_iod_destroy+0xc4 > > #5 0xffffffff81a12eea at smb_vc_free+0x1a > > #6 0xffffffff81a13e24 at sdp_trydestroy+0xb4 > > #7 0xffffffff81a1cb36 at smbfs_unmount+0xd6 > > #8 0xffffffff809d9e84 at dounmount+0x524 > > #9 0xffffffff809d9936 at sys_unmount+0x3c6 > > #10 0xffffffff80d42235 at amd64_syscall+0x265 > > #11 0xffffffff80d25cfb at Xfast_syscall+0xfb > > Uptime: 1d21h59m0s > > Dumping 191 out of 999 > > MB:..9%..17%..26%..34%..42%..51%..67%..76%..84%..92% > > > > Here are the stackframes with line numbers: > > > > (kgdb) frame 0 > > #0 __curthread () at ./machine/pcpu.h:219 > > 219 __asm("movq %%gs:%1,%0" : "=r" (td) > > (kgdb) frame 1 > > #1 doadump (textdump=) at > > /usr/src/sys/kern/kern_shutdown.c:263 > > 263 dumptid = curthread->td_tid; > > (kgdb) frame 2 > > #2 0xffffffff8093b5f2 in kern_reboot (howto=260) at > > /usr/src/sys/kern/kern_shutdown.c:451 > > 451 doadump(TRUE); > > (kgdb) frame 3 > > #3 0xffffffff8093bae5 in vpanic (fmt=, ap= > out>) at /usr/src/sys/kern/kern_shutdown.c:758 > > 758 kern_reboot(bootopt); > > (kgdb) frame 4 > > #4 0xffffffff8093b979 in kassert_panic (fmt=0xffffffff80e931ef > > "Assertion %s failed at %s:%d") at /usr/src/sys/kern/kern_shutdown.c:646 > > 646 vpanic(fmt, ap); > > (kgdb) frame 5 > > #5 0xffffffff80921c47 in _mtx_destroy (c=0xfffff80002db5490) at > > /usr/src/sys/kern/kern_mutex.c:955 > > 955 MPASS(mtx_unowned(m)); > > (kgdb) frame 6 > > #6 0xffffffff81a1c174 in smb_iod_destroy (iod=0xfffff80002db5400) at > > /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733 > > 733 smb_sl_destroy(&iod->iod_evlock); > > (kgdb) frame 7 > > #7 0xffffffff81a12eea in smb_vc_free (cp=0xfffff80002933000) at > > /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499 > > 499 smb_iod_destroy(vcp->vc_iod); > > (kgdb) frame 8 > > #8 0xffffffff81a13e24 in sdp_trydestroy (sdp=0xfffff80002904140) at > > /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166 > > 166 smb_vc_rele(vcp, scred); > > (kgdb) frame 9 > > #9 0xffffffff81a1cb96 in smbfs_unmount (mp=0xfffff80015226000, > > mntflags=) at > > /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297 > > 297 sdp_trydestroy(dev); > > (kgdb) frame 10 > > #10 0xffffffff809d9e84 in dounmount (mp=0xfffff80015226000, > > flags=134217728, td=0xfffff800151b0940) at > > /usr/src/sys/kern/vfs_mount.c:1313 > > 1313 error = VFS_UNMOUNT(mp, flags); > > (kgdb) > > > > Let me know if you need anything else from the stackframes. > > > > > > Greetings > > Christian > > > > > > > > -- > Christian Kratzer CK Software GmbH > Email: ck@cksoft.de Wildberger Weg 24/2 > Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden > Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart > Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer > Web: http://www.cksoft.de/ > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >