From owner-freebsd-current@freebsd.org Fri Mar 17 21:31:47 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 690F3D10CC0; Fri, 17 Mar 2017 21:31:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660063.outbound.protection.outlook.com [40.107.66.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 053641192; Fri, 17 Mar 2017 21:31:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0190.CANPRD01.PROD.OUTLOOK.COM (10.165.218.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.961.17; Fri, 17 Mar 2017 21:31:43 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0961.022; Fri, 17 Mar 2017 21:31:43 +0000 From: Rick Macklem To: Larry Rosenman , "freebsd-fs@freebsd.org" CC: "freebsd-current@FreeBSD.org" Subject: Re: crash: umount_nfs: Current Thread-Topic: crash: umount_nfs: Current Thread-Index: AQHSnf9DKC+Zb1AjDkCvB/ns2F6CGqGYEmLZgAAQyoCAAWs93w== Date: Fri, 17 Mar 2017 21:31:43 +0000 Message-ID: References: <20170316024433.qiujcewz5bclbgq5@borg.lerctr.org> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: lerctr.org; dkim=none (message not signed) header.d=none;lerctr.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-office365-filtering-correlation-id: e26a0004-4c66-43c7-5009-08d46d7d021a x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:YTXPR01MB0190; x-microsoft-exchange-diagnostics: 1; YTXPR01MB0190; 7:FIsWxWNnHsxXkJ5zO7jj34uNQVWqAZjp7h3M0Z5hy2dIb7seDYWemgfD6xvo7RHhfGajW3Tx9Qn7mdvzu667TLfaNnfubeelSSZEKPzpF9motQlGBIDp6ZPbe+IAosfOVJ70dxULh7uF+2kUzbPJb2Ny3jq9veK4kCN9weIy3NYTgD8zmYjA+TOkrdrWn9INS9V3Oh/Ih5ZXDPnO00eWIbe09U9TjpYub/ssA0CQZ6+gUZCyVm0bDHuPvTALjMUJHQaLCx5oV7NCfzjX3G68JSQAXv+w3oCK56Lp3VQbk5GwTfeXiUVdFtkRfTnP/3WgzX1zGyKD77w3A7HtHyLZUw== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(250305191791016)(158342451672863)(22074186197030)(209352067349851)(75325880899374)(211171220733660); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6041248)(20161123555025)(20161123564025)(20161123560025)(20161123562025)(20161123558025)(6072148); SRVR:YTXPR01MB0190; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0190; x-forefront-prvs: 0249EFCB0B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39450400003)(24454002)(377454003)(252514010)(77096006)(122556002)(7696004)(8936002)(74316002)(6506006)(2501003)(305945005)(229853002)(3280700002)(2900100001)(53546008)(4326008)(102836003)(3660700001)(53376002)(38730400002)(6436002)(86362001)(55016002)(2906002)(5660300001)(6306002)(966004)(189998001)(575784001)(9686003)(6246003)(81166006)(76176999)(50986999)(54356999)(8676002)(74482002)(2950100002)(33656002)(2004002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0190; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2017 21:31:43.8294 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0190 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Mar 2017 21:31:47 -0000 Oops, yea I see that now. In the krpc it is very hard to tell when the data= structures (that hold the mutexes) can be safely free'd. Code like xprt_unregister() g= et called asynchronously and lock a mutex as soon as called. (The crash fixed by r313735 was a prematurely free'd mutex that xprt_unregi= ster() used, but on the server side.) I'll look at the code as see if I can figure out how to delay freeing the s= tructure without leaving it allocated "forever". (I'll admit I've been tempted to just never= free it, since the memory leak this would cause would be small enough nothing would really= notice it. And, of course for your case of shutdown, it would be harmless to just = not free it.) rick ________________________________________ From: Larry Rosenman Sent: Thursday, March 16, 2017 7:46:51 PM To: Rick Macklem; freebsd-fs@freebsd.org Cc: freebsd-current@FreeBSD.org Subject: Re: crash: umount_nfs: Current Err, I=92m at r315289=85. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 On 3/16/17, 5:51 PM, "Rick Macklem" wrote: I believe the cause of this crash was fixed by a recent commit to head r313735 (which was MFC'd to stable/11 and stable/10). rick ________________________________________ From: owner-freebsd-current@freebsd.org on behalf of Larry Rosenman Sent: Wednesday, March 15, 2017 10:44:33 PM To: freebsd-fs@freebsd.org Cc: freebsd-current@FreeBSD.org Subject: crash: umount_nfs: Current Recent current, playing with new FreeNAS Corral, client is FreeBSD -CUR= RENT. Lovely crash: borg.lerctr.org dumped core - see /var/crash/vmcore.1 Wed Mar 15 21:38:53 CDT 2017 FreeBSD borg.lerctr.org 12.0-CURRENT FreeBSD 12.0-CURRENT #11 r315289: = Tue Mar 14 20:55:36 CDT 2017 root@borg.lerctr.org:/usr/obj/usr/src/sys/= VT-LER amd64 panic: general protection fault GNU gdb (GDB) 7.12.1 [GDB v7.12.1 for FreeBSD] Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copyi= ng" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/li= b/debug//boot/kernel/kernel.debug...done. done. Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid =3D 1; apic id =3D 21 instruction pointer =3D 0x20:0xffffffff80a327ae stack pointer =3D 0x28:0xfffffe535ebb2800 frame pointer =3D 0x28:0xfffffe535ebb2830 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 3172 (umount) trap number =3D 9 panic: general protection fault cpuid =3D 1 time =3D 1489631515 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe535= ebb2440 vpanic() at vpanic+0x19c/frame 0xfffffe535ebb24c0 panic() at panic+0x43/frame 0xfffffe535ebb2520 trap_fatal() at trap_fatal+0x322/frame 0xfffffe535ebb2570 trap() at trap+0x5e/frame 0xfffffe535ebb2730 calltrap() at calltrap+0x8/frame 0xfffffe535ebb2730 --- trap 0x9, rip =3D 0xffffffff80a327ae, rsp =3D 0xfffffe535ebb2800, r= bp =3D 0xfffffe535ebb2830 --- __mtx_lock_flags() at __mtx_lock_flags+0x3e/frame 0xfffffe535ebb2830 xprt_unregister() at xprt_unregister+0x28/frame 0xfffffe535ebb2850 clnt_reconnect_destroy() at clnt_reconnect_destroy+0x38/frame 0xfffffe5= 35ebb2880 nfs_unmount() at nfs_unmount+0x182/frame 0xfffffe535ebb28d0 dounmount() at dounmount+0x5c1/frame 0xfffffe535ebb2950 sys_unmount() at sys_unmount+0x30f/frame 0xfffffe535ebb2a70 amd64_syscall() at amd64_syscall+0x55a/frame 0xfffffe535ebb2bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe535ebb2bf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip =3D 0x800872b9a, rsp = =3D 0x7fffffffde88, rbp =3D 0x7fffffffe3c0 --- Uptime: 2h43m8s Dumping 5744 out of 131005 MB:..1%..11%..21%..31%..41%..51%..61%..71%..= 81%..91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/li= b/debug//boot/kernel/zfs.ko.debug...done. done. Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from= /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. done. Reading symbols from /boot/kernel/linux.ko...Reading symbols from /usr/= lib/debug//boot/kernel/linux.ko.debug...done. done. Reading symbols from /boot/kernel/linux_common.ko...Reading symbols fro= m /usr/lib/debug//boot/kernel/linux_common.ko.debug...done. done. Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from /us= r/lib/debug//boot/kernel/if_lagg.ko.debug...done. done. Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /u= sr/lib/debug//boot/kernel/coretemp.ko.debug...done. done. Reading symbols from /boot/kernel/aesni.ko...Reading symbols from /usr/= lib/debug//boot/kernel/aesni.ko.debug...done. done. Reading symbols from /boot/kernel/filemon.ko...Reading symbols from /us= r/lib/debug//boot/kernel/filemon.ko.debug...done. done. Reading symbols from /boot/kernel/fuse.ko...Reading symbols from /usr/l= ib/debug//boot/kernel/fuse.ko.debug...done. done. Reading symbols from /boot/kernel/ichsmb.ko...Reading symbols from /usr= /lib/debug//boot/kernel/ichsmb.ko.debug...done. done. Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/= lib/debug//boot/kernel/smbus.ko.debug...done. done. Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from /usr/= lib/debug//boot/kernel/ichwd.ko.debug...done. done. Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from /usr= /lib/debug//boot/kernel/cpuctl.ko.debug...done. done. Reading symbols from /boot/kernel/cryptodev.ko...Reading symbols from /= usr/lib/debug//boot/kernel/cryptodev.ko.debug...done. done. Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /= usr/lib/debug//boot/kernel/dtraceall.ko.debug...done. done. Reading symbols from /boot/kernel/profile.ko...Reading symbols from /us= r/lib/debug//boot/kernel/profile.ko.debug...done. done. Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr= /lib/debug//boot/kernel/dtrace.ko.debug...done. done. Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbo= ls from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done. done. Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /u= sr/lib/debug//boot/kernel/systrace.ko.debug...done. done. Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/li= b/debug//boot/kernel/sdt.ko.debug...done. done. Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /u= sr/lib/debug//boot/kernel/fasttrap.ko.debug...done. done. Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/li= b/debug//boot/kernel/fbt.ko.debug...done. done. Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /us= r/lib/debug//boot/kernel/dtnfscl.ko.debug...done. done. Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /u= sr/lib/debug//boot/kernel/dtmalloc.ko.debug...done. done. Reading symbols from /boot/modules/vboxdrv.ko...(no debugging symbols f= ound)...done. Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /usr/l= ib/debug//boot/kernel/ipmi.ko.debug...done. done. Reading symbols from /boot/kernel/ipmi_linux.ko...Reading symbols from = /usr/lib/debug//boot/kernel/ipmi_linux.ko.debug...done. done. Reading symbols from /boot/kernel/hwpmc.ko...Reading symbols from /usr/= lib/debug//boot/kernel/hwpmc.ko.debug...done. done. Reading symbols from /boot/kernel/mfip.ko...Reading symbols from /usr/l= ib/debug//boot/kernel/mfip.ko.debug...done. done. Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/li= b/debug//boot/kernel/ums.ko.debug...done. done. Reading symbols from /boot/modules/vboxnetflt.ko...(no debugging symbol= s found)...done. Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /u= sr/lib/debug//boot/kernel/netgraph.ko.debug...done. done. Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /u= sr/lib/debug//boot/kernel/ng_ether.ko.debug...done. done. Reading symbols from /boot/modules/vboxnetadp.ko...(no debugging symbol= s found)...done. Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /us= r/lib/debug//boot/kernel/linux64.ko.debug...done. done. Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /= usr/lib/debug//boot/kernel/linprocfs.ko.debug...done. done. Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from /usr/= lib/debug//boot/kernel/tmpfs.ko.debug...done. done. __curthread () at ./machine/pcpu.h:232 232 __asm("movq %%gs:%1,%0" : "=3Dr" (td) (kgdb) #0 __curthread () at ./machine/pcpu.h:232 #1 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:318 #2 0xffffffff80a52c15 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:386 #3 0xffffffff80a53206 in vpanic (fmt=3D, ap=3D0xfffffe5= 35ebb2500) at /usr/src/sys/kern/kern_shutdown.c:779 #4 0xffffffff80a53253 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:710 #5 0xffffffff80ecd5c2 in trap_fatal (frame=3D0xfffffe535ebb2740, eva= =3D0) at /usr/src/sys/amd64/amd64/trap.c:801 #6 0xffffffff80eccb8e in trap (frame=3D0xfffffe535ebb2740) at /usr/src/sys/amd64/amd64/trap.c:197 #7 #8 __mtx_lock_flags (c=3D0xdeadc0dedeadc0f6, opts=3D0, file=3D0xffffffff8147e721 "/usr/src/sys/rpc/svc.c", line=3D380) at /usr/src/sys/kern/kern_mutex.c:239 #9 0xffffffff80cc14b8 in xprt_unregister (xprt=3D0xfffff8022e0fce00) at /usr/src/sys/rpc/svc.c:380 #10 0xffffffff80cbc058 in clnt_reconnect_destroy (cl=3D0xfffff80146fa39= 00) at /usr/src/sys/rpc/clnt_rc.c:500 #11 0xffffffff80969972 in nfs_unmount (mp=3D, mntflags=3D) at /usr/src/sys/fs/nfsclient/nfs_clvfso= ps.c:1704 #12 0xffffffff80b0b761 in dounmount (mp=3D0xdeadc0dedeadc0f6, flags=3D1= 34217728, td=3D0xfffff8018f2eb000) at /usr/src/sys/kern/vfs_mount.c:1388 #13 0xffffffff80b0b11f in sys_unmount (td=3D0xfffff8018f2eb000, uap=3D0xfffffe535ebb2b70) at /usr/src/sys/kern/vfs_mount.c:1215 #14 0xffffffff80ecdfea in syscallenter (td=3D0xfffff8018f2eb000, sa=3D) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #15 amd64_syscall (td=3D0xfffff8018f2eb000, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:902 #16 Can't read data for section '.eh_frame' in file '/' (kgdb) vmcore / source IS available. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg"