Date: Mon, 12 Oct 2015 08:32:36 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Christian Kratzer <ck@cksoft.de> Cc: freebsd-stable@freebsd.org, John Baldwin <jhb@freebsd.org> Subject: Re: smbfs crashes since approx. 10.1-RELEASE Message-ID: <2135054744.32546564.1444653156980.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <alpine.BSF.2.20.1510121008010.47677@noc1.cksoft.de> References: <alpine.BSF.2.20.1510051157450.16263@noc1.cksoft.de> <2148690.gx9M0ZzrG1@ralph.baldwin.cx> <alpine.BSF.2.20.1510070844030.16263@noc1.cksoft.de> <3563189.eDHDcCgW5L@ralph.baldwin.cx> <alpine.BSF.2.20.1510091107010.71292@noc1.cksoft.de> <358885214.31305796.1444518367048.JavaMail.zimbra@uoguelph.ca> <alpine.BSF.2.20.1510120946150.47677@noc1.cksoft.de> <alpine.BSF.2.20.1510121008010.47677@noc1.cksoft.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Christian Kratzer wrote:
> Hi Rick,
>
> there was also a second more recent crash in /var/crash
>
> Mon Oct 12 03:01:16 CEST 2015
>
> FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M: Sun
> Oct 11 08:37:40 CEST 2015
> ck@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC amd64
>
> panic: Assertion mtx_unowned(m) failed at
> /usr/src/sys/kern/kern_mutex.c:955
>
Oops, I screwed up. I should have looked at this panic assertion when you reported
it before. Ok, so if I understand the assertion correctly, it means that another
thread has the mutex locked. If this is correct, I'll have to take another look at
the code and figure out how to wait for these other threads to finish with the mutexes.
I do think the patch fixes the race I saw, but there must be other races in the code.
I'll take another look, but if anyone else is conversant with netsmb, feel free to
jump in, because it is all new to me.
Unfortunately, I won't have any way to do testing for the next month or so, so any
patches I do come up with will be "try this untested..".
rick
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
> panic: Assertion mtx_unowned(m) failed at
> /usr/src/sys/kern/kern_mutex.c:955
> cpuid = 3
> KDB: stack backtrace:
> #0 0xffffffff80975bb0 at kdb_backtrace+0x60
> #1 0xffffffff8093baa6 at vpanic+0x126
> #2 0xffffffff8093b979 at kassert_panic+0x139
> #3 0xffffffff80921c47 at _mtx_destroy+0x77
> #4 0xffffffff81a1c174 at smb_iod_destroy+0xc4
> #5 0xffffffff81a12eea at smb_vc_free+0x1a
> #6 0xffffffff81a13e24 at sdp_trydestroy+0xb4
> #7 0xffffffff81a1cb96 at smbfs_unmount+0xd6
> #8 0xffffffff809d9e84 at dounmount+0x524
> #9 0xffffffff809d9936 at sys_unmount+0x3c6
> #10 0xffffffff80d42235 at amd64_syscall+0x265
> #11 0xffffffff80d25cfb at Xfast_syscall+0xfb
> Uptime: 8h44m10s
> Dumping 102 out of 999 MB:..16%..32%..47%..63%..78%..94%
>
> Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
> Loaded symbols for /boot/kernel/smbfs.ko.symbols
> Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
> Loaded symbols for /boot/kernel/libiconv.ko.symbols
> Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
> Loaded symbols for /boot/kernel/libmchain.ko.symbols
> #0 doadump (textdump=<value optimized out>) at pcpu.h:219
> 219 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219
> #1 0xffffffff8093b5f2 in kern_reboot (howto=260)
> at /usr/src/sys/kern/kern_shutdown.c:451
> #2 0xffffffff8093bae5 in vpanic (fmt=<value optimized out>,
> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758
> #3 0xffffffff8093b979 in kassert_panic (fmt=<value optimized out>)
> at /usr/src/sys/kern/kern_shutdown.c:646
> #4 0xffffffff80921c47 in _mtx_destroy (c=0xfffff80002db5490)
> at /usr/src/sys/kern/kern_mutex.c:955
> #5 0xffffffff81a1c174 in smb_iod_destroy (iod=0xfffff80002db5400)
> at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733
> #6 0xffffffff81a12eea in smb_vc_free (cp=0xfffff80002933000)
> at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
> #7 0xffffffff81a13e24 in sdp_trydestroy (sdp=0xfffff80002904140)
> at /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
> #8 0xffffffff81a1cb96 in smbfs_unmount (mp=0xfffff80015226000,
> mntflags=<value optimized out>)
> at /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297
> #9 0xffffffff809d9e84 in dounmount (mp=0xfffff80015226000,
> flags=134217728,
> td=0xfffff800151b0940) at /usr/src/sys/kern/vfs_mount.c:1313
> #10 0xffffffff809d9936 in sys_unmount (td=0xfffff800151b0940,
> uap=0xfffffe003d643b80) at /usr/src/sys/kern/vfs_mount.c:1205
> #11 0xffffffff80d42235 in amd64_syscall (td=0xfffff800151b0940,
> traced=0)
> at subr_syscall.c:134
> #12 0xffffffff80d25cfb in Xfast_syscall ()
> at /usr/src/sys/amd64/amd64/exception.S:396
> #13 0x000000080089190a in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language: auto; currently minimal
> (kgdb)
>
>
> Greetings
> Christian
>
>
>
>
>
>
>
>
>
>
> On Mon, 12 Oct 2015, Christian Kratzer wrote:
>
> > Hi Rick,
> >
> > On Sat, 10 Oct 2015, Rick Macklem wrote:
> >
> >> Hi again,
> >>
> >> Attached is a semantically equivalent patch to the one I posted a few
> >> minutes ago, but I think this one is more readable.
> >>
> >> Please let me know if you get it tested, rick
> >
> > the box crashed again tonight with your patch applied. Here's the new
> > crashinfo:
> >
> > panic: Assertion mtx_unowned(m) failed at
> > /usr/src/sys/kern/kern_mutex.c:955
> >
> > GNU gdb 6.1.1 [FreeBSD]
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> > are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB. Type "show warranty" for
> > details.
> > This GDB was configured as "amd64-marcel-freebsd"...
> >
> > Unread portion of the kernel message buffer:
> > panic: Assertion mtx_unowned(m) failed at
> > /usr/src/sys/kern/kern_mutex.c:955
> > cpuid = 2
> > KDB: stack backtrace:
> > #0 0xffffffff80975bb0 at kdb_backtrace+0x60
> > #1 0xffffffff8093baa6 at vpanic+0x126
> > #2 0xffffffff8093b979 at kassert_panic+0x139
> > #3 0xffffffff80921c47 at _mtx_destroy+0x77
> > #4 0xffffffff81a1c114 at smb_iod_destroy+0xc4
> > #5 0xffffffff81a12eea at smb_vc_free+0x1a
> > #6 0xffffffff81a13e24 at sdp_trydestroy+0xb4
> > #7 0xffffffff81a1cb36 at smbfs_unmount+0xd6
> > #8 0xffffffff809d9e84 at dounmount+0x524
> > #9 0xffffffff809d9936 at sys_unmount+0x3c6
> > #10 0xffffffff80d42235 at amd64_syscall+0x265
> > #11 0xffffffff80d25cfb at Xfast_syscall+0xfb
> > Uptime: 1d21h59m0s
> > Dumping 191 out of 999
> > MB:..9%..17%..26%..34%..42%..51%..67%..76%..84%..92%
> >
> > Here are the stackframes with line numbers:
> >
> > (kgdb) frame 0
> > #0 __curthread () at ./machine/pcpu.h:219
> > 219 __asm("movq %%gs:%1,%0" : "=r" (td)
> > (kgdb) frame 1
> > #1 doadump (textdump=<optimized out>) at
> > /usr/src/sys/kern/kern_shutdown.c:263
> > 263 dumptid = curthread->td_tid;
> > (kgdb) frame 2
> > #2 0xffffffff8093b5f2 in kern_reboot (howto=260) at
> > /usr/src/sys/kern/kern_shutdown.c:451
> > 451 doadump(TRUE);
> > (kgdb) frame 3
> > #3 0xffffffff8093bae5 in vpanic (fmt=<optimized out>, ap=<optimized
> > out>) at /usr/src/sys/kern/kern_shutdown.c:758
> > 758 kern_reboot(bootopt);
> > (kgdb) frame 4
> > #4 0xffffffff8093b979 in kassert_panic (fmt=0xffffffff80e931ef
> > "Assertion %s failed at %s:%d") at /usr/src/sys/kern/kern_shutdown.c:646
> > 646 vpanic(fmt, ap);
> > (kgdb) frame 5
> > #5 0xffffffff80921c47 in _mtx_destroy (c=0xfffff80002db5490) at
> > /usr/src/sys/kern/kern_mutex.c:955
> > 955 MPASS(mtx_unowned(m));
> > (kgdb) frame 6
> > #6 0xffffffff81a1c174 in smb_iod_destroy (iod=0xfffff80002db5400) at
> > /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733
> > 733 smb_sl_destroy(&iod->iod_evlock);
> > (kgdb) frame 7
> > #7 0xffffffff81a12eea in smb_vc_free (cp=0xfffff80002933000) at
> > /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
> > 499 smb_iod_destroy(vcp->vc_iod);
> > (kgdb) frame 8
> > #8 0xffffffff81a13e24 in sdp_trydestroy (sdp=0xfffff80002904140) at
> > /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
> > 166 smb_vc_rele(vcp, scred);
> > (kgdb) frame 9
> > #9 0xffffffff81a1cb96 in smbfs_unmount (mp=0xfffff80015226000,
> > mntflags=<optimized out>) at
> > /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297
> > 297 sdp_trydestroy(dev);
> > (kgdb) frame 10
> > #10 0xffffffff809d9e84 in dounmount (mp=0xfffff80015226000,
> > flags=134217728, td=0xfffff800151b0940) at
> > /usr/src/sys/kern/vfs_mount.c:1313
> > 1313 error = VFS_UNMOUNT(mp, flags);
> > (kgdb)
> >
> > Let me know if you need anything else from the stackframes.
> >
> >
> > Greetings
> > Christian
> >
> >
> >
>
> --
> Christian Kratzer CK Software GmbH
> Email: ck@cksoft.de Wildberger Weg 24/2
> Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden
> Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart
> Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer
> Web: http://www.cksoft.de/
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2135054744.32546564.1444653156980.JavaMail.zimbra>