From owner-freebsd-fs@freebsd.org Sun Dec 13 03:46:40 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 203A4A14917 for ; Sun, 13 Dec 2015 03:46:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1117112C1 for ; Sun, 13 Dec 2015 03:46:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBD3kdoU053920 for ; Sun, 13 Dec 2015 03:46:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205270] [zfs] Panic at shutdown in 11.0-CURRENT Date: Sun, 13 Dec 2015 03:46:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Dec 2015 03:46:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205270 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Dec 14 22:44:58 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E9F4A44C75 for ; Mon, 14 Dec 2015 22:44:58 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: from smtp.new-ukraine.org (smtp.new-ukraine.org [148.251.53.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.new-ukraine.org", Issuer "smtp.new-ukraine.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BD20E1B1F for ; Mon, 14 Dec 2015 22:44:57 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: on behalf of honored client by smtp.new-ukraine.org with ESMTP id tBEMftpF069575 for on Tue, 15 Dec 2015 00:42:01 +0200 (EET) Message-ID: <20151215004145.69572@smtp.new-ukraine.org> Date: Tue, 15 Dec 2015 00:41:45 -2200 From: "Zeus Panchenko" To: "FreeBSD Filesystems" cc: Subject: is there way to do "offline" mirror Organization: I.B.S. LLC Reply-To: "Zeus Panchenko" X-Attribution: zeus Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAFVBMVEWxsbGdnZ3U1NQTExN cXFzx8fG/v7+f8hyWAAACXUlEQVQ4jUWSwXYiIRBFi4yyhtjtWpmRdTL0ZC3TJOukDa6Rc+T/P2F eFepwtFvr8upVFVDua8mLWw6La4VIKTuMdAPOebdU55sQs3n/D1xFFPFGVGh4AHKttr5K0bS6g7N ZCge7qpVLB+f1Z2WAj2OKXwIWt/bXpdXSiu8KXbviWkHxF5td9+lg2e3xlI2SCvatK8YLfHyh9lw 15yrad8Va5eXg4Llr7QmAaC+dL9sDt9iad/DX3OKvLMBf+dm0A0QuMrTvYIevSik1IaSVvgjIHt5 lSCG2ynNRpEcBZ8cgDWk+Ns99qzsYYV3MZoppWzGtYlTO9+meG6m/g92iNO9LfQB2JZsMpoJs7QG ku2KtabRK0bZRwDLyBDvwlxTm6ZlP7qyOqLcfqtLexpDSB4M0H3I/PQy1emvjjzgK+A0LmMKl6Lq zlqzh0VGAw440F6MJd8cY0nI7wiF/fVIBGY7UNCAXy6DmfYGCLLI0wtDbVcDUMqtJLmAhLqODQAe riERAxXJ1/QYGpa0ymqyytpKC19MNXHjvFmEsfcHIrncFR4xdbYWgmfEGLCcZokpGbGj1egMR+6M 1BkNX1pDdhPcOXpAnAeLQUwQLYepgQoZVNGS61yaE8CYA7gYAcWKzwGstACY2HTFvvOwk4FXAG/a mKHni/EcA/GkOk7I0IK7UMIf3+SahU8/FJdiE7KcuWdM3MFocUDEEIX9LfJoo4xV5tnNKc3jJuSs SZWgnnhepgU1zN4Hii18yW4RwDX52CXUtk0Hqz6cHOIUkWaX8fDcB+J7y1y2xDHwjv/8Buu8Ekz6 7tXQAAAAASUVORK5CYII= X-Mailer: MH-E 8.3.1; nil; GNU Emacs 24.3.1 MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable X-NewUkraine-Agent: mailfromd (7.99.92) X-NewUkraine-URL: https://mail.prozora-kraina.org/smtp.html X-NewUkraine-VirStat: NO X-NewUkraine-VirScan: ScanPE, ScanELF, ScanOLE2, ScanMail, PhishingSignatures, ScanHTML, ScanPDF X-NewUkraine-SpamStat: NO X-NewUkraine-SpamScore: -1.000 of 3.500 X-NewUkraine-SpamKeys: AWL, BAYES_00, BUG6152_INVALID_DATE_TZ_ABSURD, INVALID_DATE_TZ_ABSURD, NO_RECEIVED, NO_RELAYS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Dec 2015 22:44:58 -0000 =2D----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 greetings, is there way to do "offline" mirror? what I mean: host A has zpool with mirror of two md(4) file backed, snapshots are made regularly host B has no network connection with host A what is the correct way to get on host B the same mirror of host A? variants I suggest: 1. regularly copy files of the zpool mirror from A to B? 2. regularly do something like: on A: zfs sent snapshots@after-last-B-host-mirror-sync > diff-file on B: cat diff-file | zfs receive 3. something else ... =2D --=20 Zeus V. Panchenko jid:zeus@im.ibs.dn.ua IT Dpt., I.B.S. LLC GMT+2 (EET) =2D----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlZvRakACgkQr3jpPg/3oyp2hgCg65rK2FfC0xTjc2x7jmwWmocU YlgAoN5+d93OMecaU7GOXXOqwYkUU0Hb =3D2bzi =2D----END PGP SIGNATURE----- From owner-freebsd-fs@freebsd.org Tue Dec 15 10:07:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCE9BA441F3 for ; Tue, 15 Dec 2015 10:07:02 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A8BE312F8 for ; Tue, 15 Dec 2015 10:07:02 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1a8mVM-0004CW-J5 for freebsd-fs@freebsd.org; Tue, 15 Dec 2015 11:06:54 +0100 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: is there way to do "offline" mirror References: <20151215004145.69572@smtp.new-ukraine.org> Date: Tue, 15 Dec 2015 11:06:47 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <20151215004145.69572@smtp.new-ukraine.org> User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: - X-Spam-Score: -1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Scan-Signature: 938925967a2432a0d8c7279c30be63be X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 10:07:03 -0000 On Tue, 15 Dec 2015 23:41:45 +0100, Zeus Panchenko wrote: > greetings, > > is there way to do "offline" mirror? > > what I mean: > > host A has zpool with mirror of two md(4) file backed, > snapshots are made regularly > > host B has no network connection with host A > > what is the correct way to get on host B the same mirror of host A? > > variants I suggest: > > 1. regularly copy files of the zpool mirror from A to B? > > 2. regularly do something like: > on A: zfs sent snapshots@after-last-B-host-mirror-sync > diff-file > on B: cat diff-file | zfs receive > > 3. something else ... Option 2 is nice and efficient. Need to properly administrate the diff-files so you don't miss anything. Bonus: you can even zfs diff the diff-file to see what changed. Regards, Ronald. From owner-freebsd-fs@freebsd.org Tue Dec 15 12:52:39 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88F81A43230 for ; Tue, 15 Dec 2015 12:52:39 +0000 (UTC) (envelope-from bengta@sics.se) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 686601BC2 for ; Tue, 15 Dec 2015 12:52:39 +0000 (UTC) (envelope-from bengta@sics.se) Received: by mailman.ysv.freebsd.org (Postfix) id 6733EA4322D; Tue, 15 Dec 2015 12:52:39 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66D73A4322C for ; Tue, 15 Dec 2015 12:52:39 +0000 (UTC) (envelope-from bengta@sics.se) Received: from mail-lf0-x22b.google.com (mail-lf0-x22b.google.com [IPv6:2a00:1450:4010:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E3A9C1BC1 for ; Tue, 15 Dec 2015 12:52:38 +0000 (UTC) (envelope-from bengta@sics.se) Received: by mail-lf0-x22b.google.com with SMTP id y184so5645862lfc.1 for ; Tue, 15 Dec 2015 04:52:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sics-se.20150623.gappssmtp.com; s=20150623; h=from:to:subject:user-agent:date:message-id:mime-version :content-type; bh=iH75G/zlfUZHTJHPK473RFsZ+RhHBZv3KovxYUJZWPc=; b=OEGb9L7gg1nqKb8lb376Ch0T/C2IQV8VVPXpXVCIQvvdUQlwBKqxMqqOl6Gs5RopvL E5r9kwBfldLJIeGuunk+Kq+X+XFLP07MW+1Jxea0bUml3NeVX64chipmUr27wUd8WY6Q KkTSYBbNaTJoC7TMvwjOsBZkXklcjUe7xni2Cul9DQc54SM0KVdQXtKFSS6Lz2Evaad1 u9scLp2/iEjIqeJ82Dnqp/GqBtFlabI4Bj6sObV01GmhZ3NG8+Nf9bi7L4ahFEO5bMbd oEADU/kBvMS6Mlu2NK81+JB2sEHC+aPPnGV+DMcoC1M8+tmWFsvtMb6Iyq8lr9yLOylP fN+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:user-agent:date:message-id :mime-version:content-type; bh=iH75G/zlfUZHTJHPK473RFsZ+RhHBZv3KovxYUJZWPc=; b=Spn8qXrqrGcY2DMBXDbvQNbCCGMoesR4LRGM3nJeBQJKy9DOPVuply83LLAzoypBph DBmVMx3oNzhE8SbhoiLVpllzeEqAK7bQoshWvL6cX0Rsp3flWhrhJXU5a8rAZpFuYy25 VTPQihyLds+jGvAVNvHYLZ69kCBICXt/m8iw5zI6eu1gB+QVCSEOiWvHVmMOaTLaoP4g TdkQMgqpI0ZcCzrg9OS4v/9mIDOC995w4dVAeXyN7pfKdSR3GPfBDGY9o2I28xlkzxs8 rLDcnIDmk58C/I0nzMVxx7CAeYFZp9R6vSxlrFFPlMo8m8pI0bdKstT7/xlXv2IXHaRY pI4Q== X-Gm-Message-State: ALoCoQlS1Nwq8cogWc27gEdfahbvC9bYykNw3sJejkpUAFQcuWhmrxA9uUIMOLbENw2gfd5nM3NmH9eKtBhbaQCbtc7/gsh7bQ== X-Received: by 10.25.37.137 with SMTP id l131mr8359609lfl.142.1450183956795; Tue, 15 Dec 2015 04:52:36 -0800 (PST) Received: from P142s.sics.se (P142s.sics.se. [193.10.66.127]) by smtp.gmail.com with ESMTPSA id zs6sm192481lbb.26.2015.12.15.04.52.35 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 15 Dec 2015 04:52:36 -0800 (PST) Received: from P142s.sics.se (localhost [127.0.0.1]) by P142s.sics.se (8.15.2/8.15.2) with ESMTP id tBFCpxdY002670; Tue, 15 Dec 2015 13:51:59 +0100 (CET) (envelope-from bengta@P142s.sics.se) Received: (from bengta@localhost) by P142s.sics.se (8.15.2/8.15.2/Submit) id tBFCpxvD002669; Tue, 15 Dec 2015 13:51:59 +0100 (CET) (envelope-from bengta@P142s.sics.se) From: Bengt Ahlgren To: fs@freebsd.org Subject: ZFS hang in zfs_freebsd_rename User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) Date: Tue, 15 Dec 2015 13:51:59 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 12:52:39 -0000 We have a server running 9.3-REL which currenly has two quite large zfs pools: NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - It has been running without any issues for some time now. Once, just now, processes are getting stuck and impossible to kill on accessing a particular directory in the p2 pool. That pool is a 2x6 disk raidz2. One process is stuck in zfs_freebsd_rename, and other processes accessing that particular directory also get stuck. The system is now almost completely idle. Output from kgdb on the running system for that first process: Thread 651 (Thread 102157): #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, flags=) at /usr/src/sys/kern/sched_ule.c:1904 #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:485 #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, flags=524544, ilk=0xfffffe0135b604b8, wmesg=, pri=, timo=, file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at /usr/src/sys/kern/kern_lock.c:221 #4 0xffffffff80977369 in vop_stdlock (ap=) at lockmgr.h:97 #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, a=0xffffffa07f935520) at vnode_if.c:2052 #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at vnode_if.h:859 #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at /usr/src/sys/kern/vfs_subr.c:2337 #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, a=0xffffffa07f9358e0) at vnode_if.c:1522 #11 0xffffffff80996bbd in kern_renameat (td=, oldfd=, old=, newfd=-100, new=0x1826a9af00 , pathseg=) at vnode_if.h:636 #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) at subr_syscall.c:135 #13 0xffffffff80cbc907 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 ---Type to continue, or q to quit--- #14 0x0000000800cc1acc in ?? () Previous frame inner to this frame (corrupt stack?) Full procstat -kk -a and kgdb "thread apply all bt" can be found here: https://www.sics.se/~bengta/ZFS-hang/ I don't know how to produce "alltrace in ddb" as the instructions in the wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? I checked "camcontrol tags" for all the disks in the pool - all have zeroes for dev_active, devq_queued and held. Is there anything else I can check while the machine is up? I however need to restart it pretty soon. Bengt From owner-freebsd-fs@freebsd.org Tue Dec 15 14:12:57 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0E71A439EF for ; Tue, 15 Dec 2015 14:12:57 +0000 (UTC) (envelope-from lkateley@kateley.com) Received: from mail-ig0-x22c.google.com (mail-ig0-x22c.google.com [IPv6:2607:f8b0:4001:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7266F12D4 for ; Tue, 15 Dec 2015 14:12:57 +0000 (UTC) (envelope-from lkateley@kateley.com) Received: by mail-ig0-x22c.google.com with SMTP id to4so30975724igc.0 for ; Tue, 15 Dec 2015 06:12:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kateley-com.20150623.gappssmtp.com; s=20150623; h=reply-to:subject:references:to:from:organization:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=aNdXZpw3yDyIoAozKQ64ntuvPgjOcoPm0hX6NV876DA=; b=yGVb5lHmLiod/4WjoY7CTi9heHKAlp8yF9+tm/E0SZU8VNPIfQC7VmrR38F4yd0Aof Fc/l3z6sOn+afC78yuT6K+OLZaWdWCj+ZCXezIbnrcwhg7rOoB5gr+5nP8NvSmkHIzhf 6Di1E4BNKAbmYCxpM5ijYvABHISd/fcrACZ73RwKD1yQX0jfSMLVyyH2S093Twzk+fxd 3wWvJOpVyewWkz71mCLoebQeyrLxjcg9qf0aqlg+aTc0PCggqcvhXIDzQFprC7xyjsaJ 1P3dlFuMi0vHMut/NBN5tmwG71S41onNtG0n3+9GLH5SD+dot6dn6HRFN1eHDaIi/F5S gjMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:reply-to:subject:references:to:from:organization :message-id:date:user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=aNdXZpw3yDyIoAozKQ64ntuvPgjOcoPm0hX6NV876DA=; b=Paj++KZvquvVY0z+WqcH5ryMEWg8BltLbY3dnUWpvFzVEwvDL7MDZIuAKn8pMFmoFY OgOx22ayj4OlsP3G7IrCaC6uRP0/9Mdey9ZiLrY9n8JhptB/ldNY4jy7O6XX5NJ1evAy 9CGfvIWYKpZcFsR4HysMKaMYmUouALVSIBYfbp3lkYx8FDDCd4djjiKTaRc+JT1fzAtG l/D2sJX2WI/dG42/2NX+wqh0vlF3TxR0LfXPrHsLFzM9FXToHDKHgTTB59+Eb9u6lqcl wzFqYP71H4vYSGX13ILE5KOGYt5hApKxcKwEEfk4g8+l1/RcnrL01JooX2anYnK5S5Yk hU2g== X-Gm-Message-State: ALoCoQmp20hxBbRETI9SrYvNfUof6LQgUQMPpc6A/7pOzn5deOhqaT0SF9RtGKc5O0SB5d6kIWvfjlBHN/KnPSdxaV26zTaTBw== X-Received: by 10.50.83.102 with SMTP id p6mr3929931igy.71.1450188776562; Tue, 15 Dec 2015 06:12:56 -0800 (PST) Received: from Kateleyco-iMac.local ([63.231.252.189]) by smtp.googlemail.com with ESMTPSA id k198sm1091705ioe.38.2015.12.15.06.12.55 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 15 Dec 2015 06:12:55 -0800 (PST) Reply-To: linda@kateley.com Subject: Re: is there way to do "offline" mirror References: <20151215004145.69572@smtp.new-ukraine.org> To: Zeus Panchenko , FreeBSD Filesystems From: Linda Kateley Organization: Kateley Company Message-ID: <56701FE7.9090805@kateley.com> Date: Tue, 15 Dec 2015 08:12:55 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <20151215004145.69572@smtp.new-ukraine.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 14:12:57 -0000 There is also a #zpool split poolname diska diskb That will take half the mirror off and can be imported into hostb lk On 12/15/15 4:41 PM, Zeus Panchenko wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > greetings, > > is there way to do "offline" mirror? > > what I mean: > > host A has zpool with mirror of two md(4) file backed, > snapshots are made regularly > > host B has no network connection with host A > > what is the correct way to get on host B the same mirror of host A? > > variants I suggest: > > 1. regularly copy files of the zpool mirror from A to B? > > 2. regularly do something like: > on A: zfs sent snapshots@after-last-B-host-mirror-sync > diff-file > on B: cat diff-file | zfs receive > > 3. something else ... > > - -- > Zeus V. Panchenko jid:zeus@im.ibs.dn.ua > IT Dpt., I.B.S. LLC GMT+2 (EET) > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iEYEARECAAYFAlZvRakACgkQr3jpPg/3oyp2hgCg65rK2FfC0xTjc2x7jmwWmocU > YlgAoN5+d93OMecaU7GOXXOqwYkUU0Hb > =2bzi > -----END PGP SIGNATURE----- > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Dec 15 14:26:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE082A482B2 for ; Tue, 15 Dec 2015 14:26:02 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67CD11A60 for ; Tue, 15 Dec 2015 14:26:02 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x235.google.com with SMTP id p66so93402787wmp.0 for ; Tue, 15 Dec 2015 06:26:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=+A9sDo4JBIoaaapFl6pMBhL1cJYILhPM7ps1xin0438=; b=YcoeCuuMg5WILDNIwBgcg+zlA90l8gNp4T5bn8WF6ICYY0AeUkWZ8phSye0t8PGXgh tUG+2Bt1WwkyCq5Z6oa4oB+jSaAW72ODVYgsCaEjcTVTFsD34/SUgTejnJ7RbuxRolQB nDWzW5/4BHhWcGtJrMO7lARazDp3r33miypaoJ2QtSzPprVH5dJimnOVBK9gQiyA3Qx9 8CVwzMMynUM+Nik0mmvv36xl64Kwg2+X7PpShCTWcg6hVUMpKuqcMD/IGlCELDHNx70W 9XE3a+4w8FV+p2rH8sgyIdhTM5CPk/t/b7D+5BXJfP2Gg7FiKoivG08daQY/fp7Aqru6 3Pdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=+A9sDo4JBIoaaapFl6pMBhL1cJYILhPM7ps1xin0438=; b=Qx7zfy2exoGI65vmz1/uPgtvNRpNW2ATQ2KuxEzCaM9Hdn+4PnPtAFw4AgcK3csSFd 9Aj4Wr567abw40LzqAYt3Yb2C7o13A3F84c1DFJeCtfz/i7bzQCxkhQlJTV7OjzwdINT fER+sjGbjEgBH+EgGsaR0uOVtTqQSxKOTFIF2M8WawLG+HmImzs/NHV6WhmxQCf1niQk 8d4PL9TEJH9XFbP5+DLa/Sf50UFzm4qTy/ukuKVKvrj101rB7ShXQvsUUorSVJ1/wxhh l5OCZq0WWy+r4NDgFANGYlXk9SQ+HqgqkAzar1/yiqiLOrSVeXz8cjJmom4dHydU0vX0 2Jrg== X-Gm-Message-State: ALoCoQnJolBdNKPKqiWCpsM5zlZqmKKJfClsm6tgfDK9qXgcMWWSSmy5EKawvsBz4+2xRL0OoUAR9pUWBOY1G+xY0V1vgL1E8A== X-Received: by 10.194.88.130 with SMTP id bg2mr44211723wjb.162.1450189560464; Tue, 15 Dec 2015 06:26:00 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id xs9sm1621538wjc.43.2015.12.15.06.25.59 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 15 Dec 2015 06:25:59 -0800 (PST) Subject: Re: ZFS hang in zfs_freebsd_rename To: freebsd-fs@freebsd.org References: From: Steven Hartland Message-ID: <567022FB.1010508@multiplay.co.uk> Date: Tue, 15 Dec 2015 14:26:03 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 14:26:02 -0000 Not a surprise in 9.x unfortunately, try upgrading to 10.x On 15/12/2015 12:51, Bengt Ahlgren wrote: > We have a server running 9.3-REL which currenly has two quite large zfs > pools: > > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - > p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - > > It has been running without any issues for some time now. Once, just > now, processes are getting stuck and impossible to kill on accessing a > particular directory in the p2 pool. That pool is a 2x6 disk raidz2. > > One process is stuck in zfs_freebsd_rename, and other processes > accessing that particular directory also get stuck. The system is now > almost completely idle. > > Output from kgdb on the running system for that first process: > > Thread 651 (Thread 102157): > #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, flags=) > at /usr/src/sys/kern/sched_ule.c:1904 > #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:485 > #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 > #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, flags=524544, ilk=0xfffffe0135b604b8, > wmesg=, pri=, timo=, > file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at /usr/src/sys/kern/kern_lock.c:221 > #4 0xffffffff80977369 in vop_stdlock (ap=) at lockmgr.h:97 > #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, a=0xffffffa07f935520) at vnode_if.c:2052 > #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, > file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at vnode_if.h:859 > #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at /usr/src/sys/kern/vfs_subr.c:2337 > #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) > at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 > #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) > at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 > #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, a=0xffffffa07f9358e0) at vnode_if.c:1522 > #11 0xffffffff80996bbd in kern_renameat (td=, oldfd=, > old=, newfd=-100, new=0x1826a9af00 , > pathseg=) at vnode_if.h:636 > #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) at subr_syscall.c:135 > #13 0xffffffff80cbc907 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 > ---Type to continue, or q to quit--- > #14 0x0000000800cc1acc in ?? () > Previous frame inner to this frame (corrupt stack?) > > Full procstat -kk -a and kgdb "thread apply all bt" can be found here: > > https://www.sics.se/~bengta/ZFS-hang/ > > I don't know how to produce "alltrace in ddb" as the instructions in the > wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? > > I checked "camcontrol tags" for all the disks in the pool - all have > zeroes for dev_active, devq_queued and held. > > Is there anything else I can check while the machine is up? I however > need to restart it pretty soon. > > Bengt > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Dec 15 14:52:53 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7184A443E4 for ; Tue, 15 Dec 2015 14:52:53 +0000 (UTC) (envelope-from bengta@sics.se) Received: from mail-lb0-x235.google.com (mail-lb0-x235.google.com [IPv6:2a00:1450:4010:c04::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C6771F2E for ; Tue, 15 Dec 2015 14:52:53 +0000 (UTC) (envelope-from bengta@sics.se) Received: by mail-lb0-x235.google.com with SMTP id u9so7181263lbp.2 for ; Tue, 15 Dec 2015 06:52:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sics-se.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version:content-type; bh=pEFaS1lMaRtPNvMyxprp229I7t2KCU5JRYeV0O4bHqI=; b=xYouEZvPM5DgibyGv8SGALt7NJOGlYeiBnk0M6N1nwVFLx8ncTCE1JsQWhBxigJuY8 2F+sm3cM4lLW+ti23EQMF6ix4DDEfQ3tr+BTmjx3cKGt2+L8FuvMma+SBjU07lQEmkjW ysD1fsfF3/OKun5cX4oRjA/4FW97/EVtXtHANqhBGiHZ7+OaHXtyo8Bar7JcCY9Nat76 fblj2VS4cfsFTAI/FU3yVKSde58/cW4RkPw5DItuIfZk2E9DI/wmCyDezUFb0IPI8rpv Hy1LCqOsVErc3bFkvVcj13+FyrFTzoa3VCdyCeu7wBW3WxN6U03GwWBCr+PskIeG2t87 WoVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version:content-type; bh=pEFaS1lMaRtPNvMyxprp229I7t2KCU5JRYeV0O4bHqI=; b=RdO7Hq3gbM2RRAMGjF6xc8lHqek0BldwqoLU8usMOVt7rZxfZO9lzbmIOZlhgtj0Fy Yu+mEtYMkHGlYf0FGLb6zEGQkqbKovSnXSAu1IrWSnCdaLUxtTBi3W0IqhBOGbVC4+Q1 USTZRMHqkUApopaN11s1JI/vVrh0iiILlHDfmlMYzT3/ZHlShZa/Lv22GIMOqU7Yoz3n 2PkATSiR4Y6Cuxf5M8cSctQ5/LwVUL/O0BCCa9YlWh5JCboXFCK98K9mJws+1TviFg8h bogJn6V7TjoiXEk7BXwD6T47abxFwB3tgox1eeW3pAwTEEgYMQ1eh71nA//xt898+Vby L2qA== X-Gm-Message-State: ALoCoQnyav9lCNnegWlbskoWDFhgp3I09BFzrdNZNkzSKYIOJnePCKLaN00JLx85cz/olvb+tt/AAKqsHnVQ4W00W5AybQXEuA== X-Received: by 10.112.158.100 with SMTP id wt4mr6891987lbb.46.1450191171200; Tue, 15 Dec 2015 06:52:51 -0800 (PST) Received: from P142s.sics.se (P142s.sics.se. [193.10.66.127]) by smtp.gmail.com with ESMTPSA id us1sm262766lbc.25.2015.12.15.06.52.50 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 15 Dec 2015 06:52:50 -0800 (PST) Received: from P142s.sics.se (localhost [127.0.0.1]) by P142s.sics.se (8.15.2/8.15.2) with ESMTP id tBFEqDp7003102; Tue, 15 Dec 2015 15:52:13 +0100 (CET) (envelope-from bengta@P142s.sics.se) Received: (from bengta@localhost) by P142s.sics.se (8.15.2/8.15.2/Submit) id tBFEqCMM003101; Tue, 15 Dec 2015 15:52:12 +0100 (CET) (envelope-from bengta@P142s.sics.se) From: Bengt Ahlgren To: Steven Hartland Cc: freebsd-fs@freebsd.org Subject: Re: ZFS hang in zfs_freebsd_rename In-Reply-To: <567022FB.1010508@multiplay.co.uk> (Steven Hartland's message of "Tue, 15 Dec 2015 14:26:03 +0000") References: <567022FB.1010508@multiplay.co.uk> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) Date: Tue, 15 Dec 2015 15:52:12 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 14:52:53 -0000 Yes, that is on the todo list... So this is likely fixed then in 10.x? Bengt Steven Hartland writes: > Not a surprise in 9.x unfortunately, try upgrading to 10.x > > On 15/12/2015 12:51, Bengt Ahlgren wrote: >> We have a server running 9.3-REL which currenly has two quite large zfs >> pools: >> >> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - >> p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - >> >> It has been running without any issues for some time now. Once, just >> now, processes are getting stuck and impossible to kill on accessing a >> particular directory in the p2 pool. That pool is a 2x6 disk raidz2. >> >> One process is stuck in zfs_freebsd_rename, and other processes >> accessing that particular directory also get stuck. The system is now >> almost completely idle. >> >> Output from kgdb on the running system for that first process: >> >> Thread 651 (Thread 102157): >> #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, flags=) >> at /usr/src/sys/kern/sched_ule.c:1904 >> #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:485 >> #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, >> pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 >> #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, flags=524544, ilk=0xfffffe0135b604b8, >> wmesg=, pri=, timo=, >> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at /usr/src/sys/kern/kern_lock.c:221 >> #4 0xffffffff80977369 in vop_stdlock (ap=) at lockmgr.h:97 >> #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, a=0xffffffa07f935520) at vnode_if.c:2052 >> #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, >> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at vnode_if.h:859 >> #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at /usr/src/sys/kern/vfs_subr.c:2337 >> #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) >> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 >> #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) >> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 >> #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, a=0xffffffa07f9358e0) at vnode_if.c:1522 >> #11 0xffffffff80996bbd in kern_renameat (td=, oldfd=, >> old=, newfd=-100, new=0x1826a9af00 , >> pathseg=) at vnode_if.h:636 >> #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) at subr_syscall.c:135 >> #13 0xffffffff80cbc907 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 >> ---Type to continue, or q to quit--- >> #14 0x0000000800cc1acc in ?? () >> Previous frame inner to this frame (corrupt stack?) >> >> Full procstat -kk -a and kgdb "thread apply all bt" can be found here: >> >> https://www.sics.se/~bengta/ZFS-hang/ >> >> I don't know how to produce "alltrace in ddb" as the instructions in the >> wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? >> >> I checked "camcontrol tags" for all the disks in the pool - all have >> zeroes for dev_active, devq_queued and held. >> >> Is there anything else I can check while the machine is up? I however >> need to restart it pretty soon. >> >> Bengt >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Dec 15 14:58:38 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2C2AA4498C for ; Tue, 15 Dec 2015 14:58:37 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A77A1642 for ; Tue, 15 Dec 2015 14:58:37 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22d.google.com with SMTP id p66so29476613wmp.1 for ; Tue, 15 Dec 2015 06:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=7QaCM02zNoYw6CAVAwgTXD70X1N/1B7oFUNk29E+zg8=; b=zWlkkYyacCl6Rui3s94b6OrE/xvgLUJ533xEOzWxA/Wf/X2NY811Q/BF3+F29chvo6 SzIZcUzDdAjc9nWsd40HbYtQykPRQ6slYGRZQtE9EvvoJzI0dYgZfenF8weiKCqjSxjb oipVySny54IZntM2G7Ne/K7CNiC1iJx8o6NPhHVMCzANPQ9lzLrs5dYzhezjXVtfQIW6 nXEpeJT6x6rhhNKGxQrL25hlKdKvAP9eGKv5HP0EIx+Ev3tEIjZB2qCrTZ1GYN9zlzYM aWAVbAWycngjRKd86xO85SUnfm3820SIgQK7LsWJmX0Ir+4cDFZ/f5sNczIdv6JI/cUf 6T1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=7QaCM02zNoYw6CAVAwgTXD70X1N/1B7oFUNk29E+zg8=; b=lV5Dy9xB9zQTfMq4hXGZmIdz1jeYTvKPHVobuMeBYZsq11Kp7U++KZPa1iLUXG7QT9 31Ym0SOcHpahe/6Mn0ePJJn1XnmiE1ABGNYdlMIcikHeyUow2kfxsE6URz0ErbjhhM4W 7WfDVGP10GkbnKjQQ9a5VV3DVa/V6DGmI7Vn+CA4dNaAJe8o5NhJfAkNX1aCJYGOPc3s 84FaVna72BQIcfngy6POWYyRFoN8/bjOwu11NW7bbhWEE1C0AJCuNoJOjDd6Bgwd5aRP tL4h/eE0xxrYd6pNdcP/C0EyYAa/m4o2LgusKUOpPCtuyvK9O4XCj/qIwUgPNVAes1W8 +Edw== X-Gm-Message-State: ALoCoQnjv9dGFXW4atiVsS74K4qBGguUMZiPzWbWOHBN6NuBd1ag6D9S+TZIBuq2DF9lLnT5Ai/OceZBiA6/agc6uA2dsF8HTQ== X-Received: by 10.28.13.138 with SMTP id 132mr5774977wmn.62.1450191516003; Tue, 15 Dec 2015 06:58:36 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id l20sm21406900wmd.20.2015.12.15.06.58.35 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 15 Dec 2015 06:58:35 -0800 (PST) Subject: Re: ZFS hang in zfs_freebsd_rename To: Bengt Ahlgren References: <567022FB.1010508@multiplay.co.uk> Cc: freebsd-fs@freebsd.org From: Steven Hartland Message-ID: <56702A9F.90702@multiplay.co.uk> Date: Tue, 15 Dec 2015 14:58:39 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 14:58:38 -0000 There have been quite a few reported issues with this some at least have been fix, but as with anything the only way to be sure is to test it. On 15/12/2015 14:52, Bengt Ahlgren wrote: > Yes, that is on the todo list... > > So this is likely fixed then in 10.x? > > Bengt > > Steven Hartland writes: > >> Not a surprise in 9.x unfortunately, try upgrading to 10.x >> >> On 15/12/2015 12:51, Bengt Ahlgren wrote: >>> We have a server running 9.3-REL which currenly has two quite large zfs >>> pools: >>> >>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>> p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - >>> p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - >>> >>> It has been running without any issues for some time now. Once, just >>> now, processes are getting stuck and impossible to kill on accessing a >>> particular directory in the p2 pool. That pool is a 2x6 disk raidz2. >>> >>> One process is stuck in zfs_freebsd_rename, and other processes >>> accessing that particular directory also get stuck. The system is now >>> almost completely idle. >>> >>> Output from kgdb on the running system for that first process: >>> >>> Thread 651 (Thread 102157): >>> #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, flags=) >>> at /usr/src/sys/kern/sched_ule.c:1904 >>> #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:485 >>> #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, >>> pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 >>> #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, flags=524544, ilk=0xfffffe0135b604b8, >>> wmesg=, pri=, timo=, >>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at /usr/src/sys/kern/kern_lock.c:221 >>> #4 0xffffffff80977369 in vop_stdlock (ap=) at lockmgr.h:97 >>> #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, a=0xffffffa07f935520) at vnode_if.c:2052 >>> #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, >>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at vnode_if.h:859 >>> #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at /usr/src/sys/kern/vfs_subr.c:2337 >>> #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) >>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 >>> #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) >>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 >>> #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, a=0xffffffa07f9358e0) at vnode_if.c:1522 >>> #11 0xffffffff80996bbd in kern_renameat (td=, oldfd=, >>> old=, newfd=-100, new=0x1826a9af00 , >>> pathseg=) at vnode_if.h:636 >>> #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) at subr_syscall.c:135 >>> #13 0xffffffff80cbc907 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 >>> ---Type to continue, or q to quit--- >>> #14 0x0000000800cc1acc in ?? () >>> Previous frame inner to this frame (corrupt stack?) >>> >>> Full procstat -kk -a and kgdb "thread apply all bt" can be found here: >>> >>> https://www.sics.se/~bengta/ZFS-hang/ >>> >>> I don't know how to produce "alltrace in ddb" as the instructions in the >>> wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? >>> >>> I checked "camcontrol tags" for all the disks in the pool - all have >>> zeroes for dev_active, devq_queued and held. >>> >>> Is there anything else I can check while the machine is up? I however >>> need to restart it pretty soon. >>> >>> Bengt >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Dec 15 15:01:48 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B5C9A44BC3 for ; Tue, 15 Dec 2015 15:01:48 +0000 (UTC) (envelope-from bengta@sics.se) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 835D61978 for ; Tue, 15 Dec 2015 15:01:47 +0000 (UTC) (envelope-from bengta@sics.se) Received: by mail-lb0-x22a.google.com with SMTP id u9so7365119lbp.2 for ; Tue, 15 Dec 2015 07:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sics-se.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version:content-type; bh=2CUBgGTuxNh+CNoVOpa8bmeh44PLtwc9TciyO+aC9+w=; b=s1lUyCd8JN5RVC9XFKneVTVcZsJkyp/8PajMY6sLj3OWzo/d04ujh4rL3JxXSZGuCh YTcVk3RsudiW2WnZ5++wzqgxo9W7/+ZbdX1Gmq65RkkSbQiAagY8m1fwe7Q7DfUHPvV6 YDoNVIgZfgzT26cUqVnNOz/s00RbwxDrdKlb+o+xYEVifPIK9vIhYLbobiP2go0/ncDf hg77Fz3Cvsu/g1Zc/rAbxytlqlz6K1Ih+UXCCSeiHVfjULahRiEnYAZ2Btf/f39V7jXi LfHHkld//GkDncRq0uh5bO14sI9lQfCFUFKcy4vgtX0hhdzMJu/esFFPjABn37wSE/0r 5uSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version:content-type; bh=2CUBgGTuxNh+CNoVOpa8bmeh44PLtwc9TciyO+aC9+w=; b=H1fZF8KfrIw/GU7KJkbwmPTx6bypDWRer4Cbfle8dmSeeC3RSzwnwJQ+BOFB6vEe3A x2WEB8st1ni0DdNhBhkD/fFBRXZPccXMtRtNUOk0CHJoNcX3p9TqXDmn+NP45fUJeAB0 62n+1MFC7juSlIQxWbOFxgFIUo4q4Um3qdgYB7ffzcnDBWM7PlouyrsEqj5n4cPCAJd5 mtxo/XJTPB8wYHIEZZs6lKDoE02VjhdxfDB0NUiZvv5BNDM4h60pSP0HqkkQ941Xl1tA ujrNB50vJbKsySiDyzwM9DFEj0Qrebiv8WarNwvCD+TJlDHUNY80u5yp0hdEL2E93pNc ySBw== X-Gm-Message-State: ALoCoQnAEk07fiX3eHUpg9Kr82gY+hP/tU2x32j42Mi2s+SrH482awocZiCUwroS4vTaweP+exUYEvxgje6m8apU4Z8Ty63oSg== X-Received: by 10.112.39.5 with SMTP id l5mr15593920lbk.101.1450191705586; Tue, 15 Dec 2015 07:01:45 -0800 (PST) Received: from P142s.sics.se (P142s.sics.se. [193.10.66.127]) by smtp.gmail.com with ESMTPSA id of8sm271088lbb.7.2015.12.15.07.01.44 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 15 Dec 2015 07:01:45 -0800 (PST) Received: from P142s.sics.se (localhost [127.0.0.1]) by P142s.sics.se (8.15.2/8.15.2) with ESMTP id tBFF18DL003186; Tue, 15 Dec 2015 16:01:08 +0100 (CET) (envelope-from bengta@P142s.sics.se) Received: (from bengta@localhost) by P142s.sics.se (8.15.2/8.15.2/Submit) id tBFF188n003185; Tue, 15 Dec 2015 16:01:08 +0100 (CET) (envelope-from bengta@P142s.sics.se) From: Bengt Ahlgren To: Steven Hartland Cc: freebsd-fs@freebsd.org Subject: Re: ZFS hang in zfs_freebsd_rename In-Reply-To: <56702A9F.90702@multiplay.co.uk> (Steven Hartland's message of "Tue, 15 Dec 2015 14:58:39 +0000") References: <567022FB.1010508@multiplay.co.uk> <56702A9F.90702@multiplay.co.uk> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) Date: Tue, 15 Dec 2015 16:01:08 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 15:01:48 -0000 OK, thanks for the advice! Bengt Steven Hartland writes: > There have been quite a few reported issues with this some at least > have been fix, but as with anything the only way to be sure is to test > it. > > On 15/12/2015 14:52, Bengt Ahlgren wrote: >> Yes, that is on the todo list... >> >> So this is likely fixed then in 10.x? >> >> Bengt >> >> Steven Hartland writes: >> >>> Not a surprise in 9.x unfortunately, try upgrading to 10.x >>> >>> On 15/12/2015 12:51, Bengt Ahlgren wrote: >>>> We have a server running 9.3-REL which currenly has two quite large zfs >>>> pools: >>>> >>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>> p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - >>>> p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - >>>> >>>> It has been running without any issues for some time now. Once, just >>>> now, processes are getting stuck and impossible to kill on accessing a >>>> particular directory in the p2 pool. That pool is a 2x6 disk raidz2. >>>> >>>> One process is stuck in zfs_freebsd_rename, and other processes >>>> accessing that particular directory also get stuck. The system is now >>>> almost completely idle. >>>> >>>> Output from kgdb on the running system for that first process: >>>> >>>> Thread 651 (Thread 102157): >>>> #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, flags=) >>>> at /usr/src/sys/kern/sched_ule.c:1904 >>>> #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:485 >>>> #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, >>>> pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 >>>> #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, flags=524544, ilk=0xfffffe0135b604b8, >>>> wmesg=, pri=, timo=, >>>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at /usr/src/sys/kern/kern_lock.c:221 >>>> #4 0xffffffff80977369 in vop_stdlock (ap=) at lockmgr.h:97 >>>> #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, a=0xffffffa07f935520) at vnode_if.c:2052 >>>> #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, >>>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) at vnode_if.h:859 >>>> #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at /usr/src/sys/kern/vfs_subr.c:2337 >>>> #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) >>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 >>>> #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) >>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 >>>> #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, a=0xffffffa07f9358e0) at vnode_if.c:1522 >>>> #11 0xffffffff80996bbd in kern_renameat (td=, oldfd=, >>>> old=, newfd=-100, new=0x1826a9af00 , >>>> pathseg=) at vnode_if.h:636 >>>> #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) at subr_syscall.c:135 >>>> #13 0xffffffff80cbc907 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 >>>> ---Type to continue, or q to quit--- >>>> #14 0x0000000800cc1acc in ?? () >>>> Previous frame inner to this frame (corrupt stack?) >>>> >>>> Full procstat -kk -a and kgdb "thread apply all bt" can be found here: >>>> >>>> https://www.sics.se/~bengta/ZFS-hang/ >>>> >>>> I don't know how to produce "alltrace in ddb" as the instructions in the >>>> wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? >>>> >>>> I checked "camcontrol tags" for all the disks in the pool - all have >>>> zeroes for dev_active, devq_queued and held. >>>> >>>> Is there anything else I can check while the machine is up? I however >>>> need to restart it pretty soon. >>>> >>>> Bengt >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Dec 15 15:34:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92D76A4827A for ; Tue, 15 Dec 2015 15:34:54 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C74514E8 for ; Tue, 15 Dec 2015 15:34:54 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x232.google.com with SMTP id p66so31297497wmp.1 for ; Tue, 15 Dec 2015 07:34:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=k0obaVQPHwrc4GEtyRecFUoqsqXEPZThZRTCiQJC0fc=; b=aM6iBRlphcYCiLazcxZzhUuzkntZZoNLkCRtRWIx+2cjcrmF5xgs1qJfUC1y78Lq79 dGfrV9xaPDMWsCt/InIkCmUGjiQ6b3SSslOe6yakUKGPiUjREsPYJChwRYU8KihXlWVJ 8HOEMXsyOEeBrNKpLjQQFZ/I96aTtxNwMwhYUtsGXVEk9m+05mRXx+GJ8Wy9LOREXsge nKHsA/Oxy2Uvy8bU9QJru27CTiOrdwfZ0JXXZL5jFVaVYY683hTOWcyBjGQzb1b3+bzz 9HEaxegJWIPsAyjPMHPYT2sF4Gikzsc7oyiOtsIq631gVik7DaGRubxZgWyievpaIWDJ pAxg== MIME-Version: 1.0 X-Received: by 10.28.104.134 with SMTP id d128mr5629527wmc.30.1450193691393; Tue, 15 Dec 2015 07:34:51 -0800 (PST) Received: by 10.28.181.213 with HTTP; Tue, 15 Dec 2015 07:34:51 -0800 (PST) In-Reply-To: References: <567022FB.1010508@multiplay.co.uk> <56702A9F.90702@multiplay.co.uk> Date: Tue, 15 Dec 2015 15:34:51 +0000 Message-ID: Subject: Re: ZFS hang in zfs_freebsd_rename From: krad To: Bengt Ahlgren Cc: Steven Hartland , FreeBSD FS Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 15:34:54 -0000 If your situation allows it goto stable as there have been lots of fixes since 10.2. It may be worth reviewing them to see if they are relevant. On 15 December 2015 at 15:01, Bengt Ahlgren wrote: > OK, thanks for the advice! > > Bengt > > Steven Hartland writes: > > > There have been quite a few reported issues with this some at least > > have been fix, but as with anything the only way to be sure is to test > > it. > > > > On 15/12/2015 14:52, Bengt Ahlgren wrote: > >> Yes, that is on the todo list... > >> > >> So this is likely fixed then in 10.x? > >> > >> Bengt > >> > >> Steven Hartland writes: > >> > >>> Not a surprise in 9.x unfortunately, try upgrading to 10.x > >>> > >>> On 15/12/2015 12:51, Bengt Ahlgren wrote: > >>>> We have a server running 9.3-REL which currenly has two quite large > zfs > >>>> pools: > >>>> > >>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > >>>> p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - > >>>> p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - > >>>> > >>>> It has been running without any issues for some time now. Once, just > >>>> now, processes are getting stuck and impossible to kill on accessing a > >>>> particular directory in the p2 pool. That pool is a 2x6 disk raidz2. > >>>> > >>>> One process is stuck in zfs_freebsd_rename, and other processes > >>>> accessing that particular directory also get stuck. The system is now > >>>> almost completely idle. > >>>> > >>>> Output from kgdb on the running system for that first process: > >>>> > >>>> Thread 651 (Thread 102157): > >>>> #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, > flags=) > >>>> at /usr/src/sys/kern/sched_ule.c:1904 > >>>> #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:485 > >>>> #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, > >>>> pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 > >>>> #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, > flags=524544, ilk=0xfffffe0135b604b8, > >>>> wmesg=, pri=, > timo=, > >>>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", > line=2337) at /usr/src/sys/kern/kern_lock.c:221 > >>>> #4 0xffffffff80977369 in vop_stdlock (ap=) at > lockmgr.h:97 > >>>> #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, > a=0xffffffa07f935520) at vnode_if.c:2052 > >>>> #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, > flags=524288, > >>>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", > line=2337) at vnode_if.h:859 > >>>> #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at > /usr/src/sys/kern/vfs_subr.c:2337 > >>>> #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) > >>>> at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 > >>>> #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap= out>) > >>>> at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 > >>>> #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, > a=0xffffffa07f9358e0) at vnode_if.c:1522 > >>>> #11 0xffffffff80996bbd in kern_renameat (td=, > oldfd=, > >>>> old=, newfd=-100, new=0x1826a9af00 reading address 0x1826a9af00: Bad address>, > >>>> pathseg=) at vnode_if.h:636 > >>>> #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, > traced=0) at subr_syscall.c:135 > >>>> #13 0xffffffff80cbc907 in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:396 > >>>> ---Type to continue, or q to quit--- > >>>> #14 0x0000000800cc1acc in ?? () > >>>> Previous frame inner to this frame (corrupt stack?) > >>>> > >>>> Full procstat -kk -a and kgdb "thread apply all bt" can be found here: > >>>> > >>>> https://www.sics.se/~bengta/ZFS-hang/ > >>>> > >>>> I don't know how to produce "alltrace in ddb" as the instructions in > the > >>>> wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? > >>>> > >>>> I checked "camcontrol tags" for all the disks in the pool - all have > >>>> zeroes for dev_active, devq_queued and held. > >>>> > >>>> Is there anything else I can check while the machine is up? I however > >>>> need to restart it pretty soon. > >>>> > >>>> Bengt > >>>> _______________________________________________ > >>>> freebsd-fs@freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Dec 15 21:32:48 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B35E2A48096 for ; Tue, 15 Dec 2015 21:32:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 4F9851262 for ; Tue, 15 Dec 2015 21:32:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:ilX2ZRDrNq2WpafIdoM5UyQJP3N1i/DPJgcQr6AfoPdwSP/4ocbcNUDSrc9gkEXOFd2CrakU1ayO6+jJYi8p39WoiDg6aptCVhsI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6kO74TNaIBjjLw09fr2zQd6MyZzvn8mJuLTtICxwzAKnZr1zKBjk5S7wjeIxxbVYF6Aq1xHSqWFJcekFjUlhJFaUggqurpzopM0r221qtvkg789NV7nhN+R9FOQATWduDmYu+ce+tQXfVRDdoTwYU34KiVxGGQXI5gr2GJDrvWz/v+t53SCcesn3Vqw1XzqlqKlxRRLikytCOSVqzGaCsdB9kq9d6DKovQB7yojYKNWWNf56f6XSVdYHQXZARsJYRmpKBcWhbN1cIfAGOLNiroL+734Hphi6CAzkUPnqwzRLgnLz9bA93PksFRnGmgcpSYFd+E/Ipcn4Yf9BGdu+y7PFmHCaN6tb X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CtBADuhXBW/61jaINeFoN2bQa9YYFjFwqFIkoCggQSAQEBAQEBAQGBCYItggcBAQEEAQEBICsgCwwEAgEIDgoCAg0ZAgInAQkmAgQIBwQBHASIDg6rdJF3AQEBAQEBAQEBAQEBAQEBAQEBARYEgQGFVYR9hCAbAQEFCRaDFoFJBY01d4hQhTmFIpInjV4CKAM4hCIgNAeDJAcXI4EIAQEB X-IronPort-AV: E=Sophos;i="5.20,434,1444708800"; d="scan'208";a="258029635" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 15 Dec 2015 16:32:41 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 2C35D15F565; Tue, 15 Dec 2015 16:32:41 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 6jMgO7VivfOV; Tue, 15 Dec 2015 16:32:40 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 021E515F55D; Tue, 15 Dec 2015 16:32:40 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id zs7AyB2iqUeo; Tue, 15 Dec 2015 16:32:39 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C9C3015F56E; Tue, 15 Dec 2015 16:32:39 -0500 (EST) Date: Tue, 15 Dec 2015 16:32:39 -0500 (EST) From: Rick Macklem To: Steven Hartland Cc: Bengt Ahlgren , freebsd-fs@freebsd.org Message-ID: <865572400.133527790.1450215159693.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <56702A9F.90702@multiplay.co.uk> References: <567022FB.1010508@multiplay.co.uk> <56702A9F.90702@multiplay.co.uk> Subject: Re: ZFS hang in zfs_freebsd_rename MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF42 (Win)/8.0.9_GA_6191) Thread-Topic: ZFS hang in zfs_freebsd_rename Thread-Index: HhrgZNowbOuAMUqLR1lHOtfW6eGjIg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 21:32:48 -0000 I'm not a ZFS guy, but I vaguely recall that renaming of snapshots can (or at least could, I don't know if it has been fixed) cause hung threads due to lock ordering issues. So, if by any chance you are renaming snapshots, you might want to avoid doing that. rick ----- Original Message ----- > There have been quite a few reported issues with this some at least have > been fix, but as with anything the only way to be sure is to test it. > > On 15/12/2015 14:52, Bengt Ahlgren wrote: > > Yes, that is on the todo list... > > > > So this is likely fixed then in 10.x? > > > > Bengt > > > > Steven Hartland writes: > > > >> Not a surprise in 9.x unfortunately, try upgrading to 10.x > >> > >> On 15/12/2015 12:51, Bengt Ahlgren wrote: > >>> We have a server running 9.3-REL which currenly has two quite large zfs > >>> pools: > >>> > >>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > >>> p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - > >>> p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - > >>> > >>> It has been running without any issues for some time now. Once, just > >>> now, processes are getting stuck and impossible to kill on accessing a > >>> particular directory in the p2 pool. That pool is a 2x6 disk raidz2. > >>> > >>> One process is stuck in zfs_freebsd_rename, and other processes > >>> accessing that particular directory also get stuck. The system is now > >>> almost completely idle. > >>> > >>> Output from kgdb on the running system for that first process: > >>> > >>> Thread 651 (Thread 102157): > >>> #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, > >>> flags=) > >>> at /usr/src/sys/kern/sched_ule.c:1904 > >>> #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at > >>> /usr/src/sys/kern/kern_synch.c:485 > >>> #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, > >>> pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 > >>> #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, > >>> flags=524544, ilk=0xfffffe0135b604b8, > >>> wmesg=, pri=, timo= >>> optimized out>, > >>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) > >>> at /usr/src/sys/kern/kern_lock.c:221 > >>> #4 0xffffffff80977369 in vop_stdlock (ap=) at > >>> lockmgr.h:97 > >>> #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, > >>> a=0xffffffa07f935520) at vnode_if.c:2052 > >>> #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, > >>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) > >>> at vnode_if.h:859 > >>> #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at > >>> /usr/src/sys/kern/vfs_subr.c:2337 > >>> #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) > >>> at > >>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 > >>> #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) > >>> at > >>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 > >>> #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, > >>> a=0xffffffa07f9358e0) at vnode_if.c:1522 > >>> #11 0xffffffff80996bbd in kern_renameat (td=, > >>> oldfd=, > >>> old=, newfd=-100, new=0x1826a9af00 >>> reading address 0x1826a9af00: Bad address>, > >>> pathseg=) at vnode_if.h:636 > >>> #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) > >>> at subr_syscall.c:135 > >>> #13 0xffffffff80cbc907 in Xfast_syscall () at > >>> /usr/src/sys/amd64/amd64/exception.S:396 > >>> ---Type to continue, or q to quit--- > >>> #14 0x0000000800cc1acc in ?? () > >>> Previous frame inner to this frame (corrupt stack?) > >>> > >>> Full procstat -kk -a and kgdb "thread apply all bt" can be found here: > >>> > >>> https://www.sics.se/~bengta/ZFS-hang/ > >>> > >>> I don't know how to produce "alltrace in ddb" as the instructions in the > >>> wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? > >>> > >>> I checked "camcontrol tags" for all the disks in the pool - all have > >>> zeroes for dev_active, devq_queued and held. > >>> > >>> Is there anything else I can check while the machine is up? I however > >>> need to restart it pretty soon. > >>> > >>> Bengt > >>> _______________________________________________ > >>> freebsd-fs@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Dec 15 22:06:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 994B4A48597 for ; Tue, 15 Dec 2015 22:06:54 +0000 (UTC) (envelope-from bengta@sics.se) Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 382BC15B3 for ; Tue, 15 Dec 2015 22:06:54 +0000 (UTC) (envelope-from bengta@sics.se) Received: by mail-lf0-x233.google.com with SMTP id y184so16820010lfc.1 for ; Tue, 15 Dec 2015 14:06:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sics-se.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version:content-type; bh=4VuyDh0Nl2n02GCybP9E7e0HnLuOTHDfuXSjQq+3bCw=; b=CewHuVoY2nKbpsXGj1k2WDiZ9S8gXWl/Frc0UE8iCwtpCT8SjJQJ9++guoaKVkZ8j9 hB5WMHMvHnYmqsZtDMYX7znLfKii0h8lwk9CFPKaYxLy6OHS0jbXjDjlRkRI5l2fGXvT lIbcAQfhW7L6mIM9OUnZiOkgcSnqyddtuXhC9HuZhyZ/mTcTuCzOMWsot315APJNw1c4 jIqxNoQXGY2+I7Fn6WAMw8kst8I0Pd7Hj1gvCzk8ZJL9Vpk6bz+/szyJBEl34+eEdfSd fH1KzqgTOMI8ZR0mwHFxhwMc8EMK9sO3lcuDUfX4nUJvwg7q9n0BKH/2uN8z1x9mJ4i0 RmVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version:content-type; bh=4VuyDh0Nl2n02GCybP9E7e0HnLuOTHDfuXSjQq+3bCw=; b=ASl6IQ3dhYzOFt1Sm0Ks+y9RkBxutJfdvuMEKJufPTIOG2hLlwXjIfPFagc9z6VXLr JGHIhE341l7kx74I6JLAz+MeG+GnlVDYL8z0QH0ro7sCgt4m1DnHuLDeIF6dUFtgpPWX siYxjxJWKkvVlcAwFvzfxQSK7F2U7jvIflwqhqWBg5/lCT2g2d2m+ceXPZsIE5g21QaJ qAIsTFN784Lds1h62MgZ97cubD9MCTe/f0tmkXCL6iT+4tD6GF2xVUxi+uYFQKZ9/YjH elabRPgZ+R/FULVF0371QurDfFGXmdqC0GX9FxAJr4G4EWbkISN52s9ZnipsNctUjn52 ASDQ== X-Gm-Message-State: ALoCoQnCriKNthmlA91J5njRtXSUdN6WH8iqIYCDGwwJEeSOd+RNDql/I1+8m0jFBY7s1RVdOdNtHhPXIYPmeB/3/lKdChRH9w== X-Received: by 10.25.211.209 with SMTP id k200mr16823851lfg.125.1450217211922; Tue, 15 Dec 2015 14:06:51 -0800 (PST) Received: from P142s.sics.se (h139n3-u-d1.ias.bredband.telia.com. [90.228.197.139]) by smtp.gmail.com with ESMTPSA id d130sm504151lfe.18.2015.12.15.14.06.50 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 15 Dec 2015 14:06:51 -0800 (PST) Received: from P142s.sics.se (localhost [127.0.0.1]) by P142s.sics.se (8.15.2/8.15.2) with ESMTP id tBFM6ENu002134; Tue, 15 Dec 2015 23:06:14 +0100 (CET) (envelope-from bengta@P142s.sics.se) Received: (from bengta@localhost) by P142s.sics.se (8.15.2/8.15.2/Submit) id tBFM6Dbk002133; Tue, 15 Dec 2015 23:06:13 +0100 (CET) (envelope-from bengta@P142s.sics.se) From: Bengt Ahlgren To: Rick Macklem Cc: Steven Hartland , freebsd-fs@freebsd.org Subject: Re: ZFS hang in zfs_freebsd_rename In-Reply-To: <865572400.133527790.1450215159693.JavaMail.zimbra@uoguelph.ca> (Rick Macklem's message of "Tue, 15 Dec 2015 16:32:39 -0500 (EST)") References: <567022FB.1010508@multiplay.co.uk> <56702A9F.90702@multiplay.co.uk> <865572400.133527790.1450215159693.JavaMail.zimbra@uoguelph.ca> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) Date: Tue, 15 Dec 2015 23:06:13 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 22:06:54 -0000 The pool has a few snapshots, but no renaming of them took place any time recently. This was renaming of a file. Bengt Rick Macklem writes: > I'm not a ZFS guy, but I vaguely recall that renaming of snapshots > can (or at least could, I don't know if it has been fixed) cause > hung threads due to lock ordering issues. > > So, if by any chance you are renaming snapshots, you might want to > avoid doing that. > > rick > > ----- Original Message ----- >> There have been quite a few reported issues with this some at least have >> been fix, but as with anything the only way to be sure is to test it. >> >> On 15/12/2015 14:52, Bengt Ahlgren wrote: >> > Yes, that is on the todo list... >> > >> > So this is likely fixed then in 10.x? >> > >> > Bengt >> > >> > Steven Hartland writes: >> > >> >> Not a surprise in 9.x unfortunately, try upgrading to 10.x >> >> >> >> On 15/12/2015 12:51, Bengt Ahlgren wrote: >> >>> We have a server running 9.3-REL which currenly has two quite large zfs >> >>> pools: >> >>> >> >>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> >>> p1 18.1T 10.7T 7.38T 59% 1.00x ONLINE - >> >>> p2 43.5T 29.1T 14.4T 66% 1.00x ONLINE - >> >>> >> >>> It has been running without any issues for some time now. Once, just >> >>> now, processes are getting stuck and impossible to kill on accessing a >> >>> particular directory in the p2 pool. That pool is a 2x6 disk raidz2. >> >>> >> >>> One process is stuck in zfs_freebsd_rename, and other processes >> >>> accessing that particular directory also get stuck. The system is now >> >>> almost completely idle. >> >>> >> >>> Output from kgdb on the running system for that first process: >> >>> >> >>> Thread 651 (Thread 102157): >> >>> #0 sched_switch (td=0xfffffe0b14059920, newtd=0xfffffe001633e920, >> >>> flags=) >> >>> at /usr/src/sys/kern/sched_ule.c:1904 >> >>> #1 0xffffffff808f4604 in mi_switch (flags=260, newtd=0x0) at >> >>> /usr/src/sys/kern/kern_synch.c:485 >> >>> #2 0xffffffff809308e2 in sleepq_wait (wchan=0xfffffe0135b60488, >> >>> pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:618 >> >>> #3 0xffffffff808cf922 in __lockmgr_args (lk=0xfffffe0135b60488, >> >>> flags=524544, ilk=0xfffffe0135b604b8, >> >>> wmesg=, pri=, timo=> >>> optimized out>, >> >>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) >> >>> at /usr/src/sys/kern/kern_lock.c:221 >> >>> #4 0xffffffff80977369 in vop_stdlock (ap=) at >> >>> lockmgr.h:97 >> >>> #5 0xffffffff80dd4a04 in VOP_LOCK1_APV (vop=0xffffffff813e8160, >> >>> a=0xffffffa07f935520) at vnode_if.c:2052 >> >>> #6 0xffffffff80998c17 in _vn_lock (vp=0xfffffe0135b603f0, flags=524288, >> >>> file=0xffffffff80f0d782 "/usr/src/sys/kern/vfs_subr.c", line=2337) >> >>> at vnode_if.h:859 >> >>> #7 0xffffffff8098b621 in vputx (vp=0xfffffe0135b603f0, func=1) at >> >>> /usr/src/sys/kern/vfs_subr.c:2337 >> >>> #8 0xffffffff81ac7955 in zfs_rename_unlock (zlpp=0xffffffa07f9356b8) >> >>> at >> >>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3609 >> >>> #9 0xffffffff81ac8c72 in zfs_freebsd_rename (ap=) >> >>> at >> >>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4039 >> >>> #10 0xffffffff80dd4f04 in VOP_RENAME_APV (vop=0xffffffff81b47d40, >> >>> a=0xffffffa07f9358e0) at vnode_if.c:1522 >> >>> #11 0xffffffff80996bbd in kern_renameat (td=, >> >>> oldfd=, >> >>> old=, newfd=-100, new=0x1826a9af00 > >>> reading address 0x1826a9af00: Bad address>, >> >>> pathseg=) at vnode_if.h:636 >> >>> #12 0xffffffff80cd228a in amd64_syscall (td=0xfffffe0b14059920, traced=0) >> >>> at subr_syscall.c:135 >> >>> #13 0xffffffff80cbc907 in Xfast_syscall () at >> >>> /usr/src/sys/amd64/amd64/exception.S:396 >> >>> ---Type to continue, or q to quit--- >> >>> #14 0x0000000800cc1acc in ?? () >> >>> Previous frame inner to this frame (corrupt stack?) >> >>> >> >>> Full procstat -kk -a and kgdb "thread apply all bt" can be found here: >> >>> >> >>> https://www.sics.se/~bengta/ZFS-hang/ >> >>> >> >>> I don't know how to produce "alltrace in ddb" as the instructions in the >> >>> wiki says. It runs the GENERIC kernel, so perhaps it isn't possible? >> >>> >> >>> I checked "camcontrol tags" for all the disks in the pool - all have >> >>> zeroes for dev_active, devq_queued and held. >> >>> >> >>> Is there anything else I can check while the machine is up? I however >> >>> need to restart it pretty soon. >> >>> >> >>> Bengt From owner-freebsd-fs@freebsd.org Wed Dec 16 13:13:09 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B3DDA48A36 for ; Wed, 16 Dec 2015 13:13:09 +0000 (UTC) (envelope-from bengta@sics.se) Received: from mail-lf0-x22e.google.com (mail-lf0-x22e.google.com [IPv6:2a00:1450:4010:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5462E03 for ; Wed, 16 Dec 2015 13:13:09 +0000 (UTC) (envelope-from bengta@sics.se) Received: by mail-lf0-x22e.google.com with SMTP id y184so28802342lfc.1 for ; Wed, 16 Dec 2015 05:13:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sics-se.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version:content-type; bh=5Kxkj2VKOx0kXbf3W7xj0nQkiItTm0BmAq35sb0NrUo=; b=VUkY02bMisV1lmxXD2vwIt+uVYXO5IjWdoUhIiR9ckNK3b52zkI6rFjdJXmyhJom8P QEifH/LqocN2p+NRndFBHQPseZU+k5IvB30hfsgCcu8qZCOFgwY0yu4rSA76ELPahNWb bhjzjs+dt//N+6KtMSuvhAiEWuGUCa38I6CZQTJmz/8vcW+JAnX5k+gwhvYQcw9u8RrK bL30yT/D/NZsdAjiXw3+9nF8k9bhfDn5gJ7b5/ZrnJJSRaaqaj7yi38gfdjs0LFFmpaq EKDjgY3v8ZceolR2mhjnKI0kHEdlOxsTnTB5z6IZx/KKWr5M+MVTtaG13aQnEIhmcNBs BIBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version:content-type; bh=5Kxkj2VKOx0kXbf3W7xj0nQkiItTm0BmAq35sb0NrUo=; b=eQRjDdmoKusyW1cNc058klOIkq4uhdjVMsDxPKKonpvQjuLFHNhxjTLi3TkVr6ZKO4 1OV2/HbPhb5RtHJ8L2tmunnf/NaFHEQTNBnMQWhjRplWEOvQzq28Cff5FxlibMoTYGwr k1eGNa+/jDswxFODjhwGKqzurXJptE2AV5u+f0gjz3U23WS1NIKLpZDA5Q5h+ROqbK+x DEu3NG6vpeDOI4n8CcGqdSP3qoDcQyyNnOEzGGYTTcqkKNMpqRKJRmtGwjzI5zU5TMIn HofeLo1OJEsGpiqcbJAZnPwjQTlbjJKMmmvERw0UosEGCIZlIdD/C0FN0B97Iklms46T dOpA== X-Gm-Message-State: ALoCoQkRYgJ2xFy5uLcPPyXtdDDfiVZ7qxNZ7wm07esjOBPD2Z0OWNgYkIh1kUHJH1NXRzAwml6qM1tBxbSJMG8iJKrv2jXNsQ== X-Received: by 10.25.147.84 with SMTP id v81mr18371338lfd.119.1450268108047; Wed, 16 Dec 2015 04:15:08 -0800 (PST) Received: from P142s.sics.se (P142s.sics.se. [193.10.66.127]) by smtp.gmail.com with ESMTPSA id xt2sm917413lbb.47.2015.12.16.04.15.07 (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 16 Dec 2015 04:15:07 -0800 (PST) Received: from P142s.sics.se (localhost [127.0.0.1]) by P142s.sics.se (8.15.2/8.15.2) with ESMTP id tBGCEUMZ003758; Wed, 16 Dec 2015 13:14:30 +0100 (CET) (envelope-from bengta@P142s.sics.se) Received: (from bengta@localhost) by P142s.sics.se (8.15.2/8.15.2/Submit) id tBGCET9Y003757; Wed, 16 Dec 2015 13:14:29 +0100 (CET) (envelope-from bengta@P142s.sics.se) From: Bengt Ahlgren To: Andriy Gapon Cc: Steven Hartland , freebsd-fs@FreeBSD.org Subject: Re: ZFS hang in zfs_freebsd_rename In-Reply-To: <56714344.1070908@FreeBSD.org> (Andriy Gapon's message of "Wed, 16 Dec 2015 12:56:04 +0200") References: <567022FB.1010508@multiplay.co.uk> <56714344.1070908@FreeBSD.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) Date: Wed, 16 Dec 2015 13:14:29 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Dec 2015 13:13:09 -0000 Andriy Gapon writes: > On 15/12/2015 16:52, Bengt Ahlgren wrote: >> So this is likely fixed then in 10.x? > > No and neither in head / current. We haven't had any fixes in ZFS<->VFS code > for quite a long while. > And this is a FreeBSD-specific problem. zfs_rename() does not behave as > VOP_RENAME should do with respect to the vnode locking. OK, good to know. I assume this means that it would be beneficial to extract more information if this deadlock happens again, regardless of whether or not I have had the time to upgrade to 10.x. Bengt From owner-freebsd-fs@freebsd.org Wed Dec 16 13:18:23 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68F10A490BC for ; Wed, 16 Dec 2015 13:18:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 75A2A643B5 for ; Wed, 16 Dec 2015 13:18:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA25497; Wed, 16 Dec 2015 12:57:26 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1a99lu-000On3-LF; Wed, 16 Dec 2015 12:57:26 +0200 Subject: Re: ZFS hang in zfs_freebsd_rename To: Bengt Ahlgren , Steven Hartland References: <567022FB.1010508@multiplay.co.uk> Cc: freebsd-fs@FreeBSD.org From: Andriy Gapon Message-ID: <56714344.1070908@FreeBSD.org> Date: Wed, 16 Dec 2015 12:56:04 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Dec 2015 13:18:23 -0000 On 15/12/2015 16:52, Bengt Ahlgren wrote: > So this is likely fixed then in 10.x? No and neither in head / current. We haven't had any fixes in ZFS<->VFS code for quite a long while. And this is a FreeBSD-specific problem. zfs_rename() does not behave as VOP_RENAME should do with respect to the vnode locking. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Dec 16 14:13:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ED78A488C2 for ; Wed, 16 Dec 2015 14:13:41 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: from mail-qg0-x231.google.com (mail-qg0-x231.google.com [IPv6:2607:f8b0:400d:c04::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 507E0122B for ; Wed, 16 Dec 2015 14:13:41 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: by mail-qg0-x231.google.com with SMTP id 103so34514191qgi.3 for ; Wed, 16 Dec 2015 06:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kraus-haus-org.20150623.gappssmtp.com; s=20150623; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=yRU/+ZorCc9ecwsS3Eoxnxx8s78yeJkz7jkjPijiyEM=; b=mE048lz75bqaEZgZzPaEVoqZjTnzhgHPwxfn+UG9RSl9HjZf/SIzUppgUPSzTwK2cZ tqC4U5E9Xk6YYy1RZfLEqebKHag0bkt9ECbCeocCcM50ad4Z/4Vh2ELFX2ieXYzMy3Tw n86PuAQPaPudGvrXpJayqyeiuDzQuoZylKX2yzK0M2jl5imzmhZUivB4xnFqHrnioVxn xInCBGxL97kRyGI9zQQcKDtJqwziKKEuqo3oPmeBXOFESDMdCQM5Le8P2UzNpQCzyhB7 SdRdGK2gyyoGIJpxKsNyX1/JwctxJeS4yp6/XMkBoa8N7fpre5fQKKhUKuFrxOliLE05 hB5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:content-transfer-encoding:message-id:references:to; bh=yRU/+ZorCc9ecwsS3Eoxnxx8s78yeJkz7jkjPijiyEM=; b=AvA5sS4n0sHqFFdyfUfPSzhSk7xNOp2MdHeF82YABQfckuQIhtnOrx8Hn61vMCb0UT 3ZA4/HRUVxiwCTm5IkebuiUsnbZnoMBy56UaNzH7A7gXR6Xh/NfbXbrWvWqtTTDBe58n T/dUODaz7v2Z3ERsM7yPOuaTY5HQjN6eI31mu9fOmJM92Mss0HwuEpldv5M/ko5lfsz1 jqu475EY+9PmzNqCx6ZVyDF0MOAtKBRH+YA0q3+0j1FZPjIKfmfsbutBs05ox7c/iS05 BX4JDzePrx9n3BzBEMSpvyuqT0yqh/a3OF0Kwabry6XlBPejU5vA05ujgARWl49pq4HG +P+g== X-Gm-Message-State: ALoCoQkIBp2iVypkaWdPoxSgQIuu8HfiwHQeB0vl19q+umNHfwl9Vr9UuvgixFm9O5DZDP6zt2k/hgHUvpCsRP1yl6O8p8FUeQ== X-Received: by 10.140.160.69 with SMTP id g66mr49620084qhg.86.1450275220196; Wed, 16 Dec 2015 06:13:40 -0800 (PST) Received: from mbp-1.thecreativeadvantage.com (mail.thecreativeadvantage.com. [96.236.20.34]) by smtp.gmail.com with ESMTPSA id q14sm2662160qkh.4.2015.12.16.06.13.38 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 16 Dec 2015 06:13:38 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: ZFS hang in zfs_freebsd_rename From: Paul Kraus In-Reply-To: <56714344.1070908@FreeBSD.org> Date: Wed, 16 Dec 2015 09:13:36 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <567022FB.1010508@multiplay.co.uk> <56714344.1070908@FreeBSD.org> To: freebsd-fs@FreeBSD.org X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Dec 2015 14:13:41 -0000 On Dec 16, 2015, at 5:56, Andriy Gapon wrote: > On 15/12/2015 16:52, Bengt Ahlgren wrote: >> So this is likely fixed then in 10.x? >=20 > No and neither in head / current. We haven't had any fixes in = ZFS<->VFS code > for quite a long while. > And this is a FreeBSD-specific problem. zfs_rename() does not behave = as > VOP_RENAME should do with respect to the vnode locking. I assume that a zpool export followed by an import using the UUID and = giving the pool a different name is a suitable workaround ? This also = assumes downtime. -- Paul Kraus paul@kraus-haus.org