From owner-freebsd-fs@freebsd.org Sun Feb 21 12:45:22 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5348EAB0FA8 for ; Sun, 21 Feb 2016 12:45:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44D251386 for ; Sun, 21 Feb 2016 12:45:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1LCjLVl037810 for ; Sun, 21 Feb 2016 12:45:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Sun, 21 Feb 2016 12:45:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: roel@qsp.nl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Feb 2016 12:45:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 roel@qsp.nl changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |roel@qsp.nl --- Comment #5 from roel@qsp.nl --- Created attachment 167247 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D167247&action= =3Dedit procstat -kk -a output output from procstat -kk -a at the moment of the deadlock --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Feb 21 12:47:26 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1608AAAF0C6 for ; Sun, 21 Feb 2016 12:47:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0797C14A0 for ; Sun, 21 Feb 2016 12:47:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1LClP9m040364 for ; Sun, 21 Feb 2016 12:47:25 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Sun, 21 Feb 2016 12:47:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: roel@qsp.nl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Feb 2016 12:47:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 --- Comment #6 from roel@qsp.nl --- We've run into this exact same issue using the zrep script. It's easily reproducable. The deadlock occurs when a "zfs rename" is executed while the= re is still a "zfs send" process running. In zrep, this occurs when a zfs receive on the remote hosts fails and zrep immediately attempts to rename a snapshot. However, the zfs send process is still running (cleaning up?) and this causes an immediate deadlock. Any zfs process run after that immediately blocks in state "zpa_namespace_lock". IO continues to work, but fails at some point in the future. I've not diagnosed the exact trigger that causes IO to seize. The only resolution is to power cycle the system. I've just attached the output to a procstat -kk -a from yesterday when I ran into this. Running on 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #1 r294572 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Feb 21 15:00:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4CA50AB0474 for ; Sun, 21 Feb 2016 15:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CD6D1084 for ; Sun, 21 Feb 2016 15:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1LF0ChC058778 for ; Sun, 21 Feb 2016 15:00:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Sun, 21 Feb 2016 15:00:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Feb 2016 15:00:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |smh@FreeBSD.org --- Comment #7 from Andriy Gapon --- (In reply to roel from comment #6) It would be interesting to run kgdb and examine spa_namespace_lock. Forcing= a crash dump for post-mortem examination is probably more useful because it w= ould be easier to answer any possible follow-up questions. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Feb 22 12:36:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24A56AB04A2 for ; Mon, 22 Feb 2016 12:36:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id E388C1964 for ; Mon, 22 Feb 2016 12:36:44 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:f561:4812:9272:f97f]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 9BA86ABAF; Mon, 22 Feb 2016 15:36:42 +0300 (MSK) Date: Mon, 22 Feb 2016 15:36:42 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <1292016778.20160222153637@serebryakov.spb.ru> To: Rick Macklem CC: freebsd-fs@freebsd.org Subject: Re: Panic in NFS client on CURRENT In-Reply-To: <796102748.572844.1456009409825.JavaMail.zimbra@uoguelph.ca> References: <56C752CD.4090203@FreeBSD.org> <1022369130.4303814.1455930123897.JavaMail.zimbra@uoguelph.ca> <56C84922.8050803@FreeBSD.org> <796102748.572844.1456009409825.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="pgp-sha512"; boundary="----------12219C0062CF59573" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 12:36:45 -0000 ------------12219C0062CF59573 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: base64 SGVsbG8gUmljaywNCg0KU3VuZGF5LCBGZWJydWFyeSAyMSwgMjAxNiwgMjowMzoyOSBBTSwg eW91IHdyb3RlOg0KDQo+IE9oLCBhbmQgaWYgeW91IGFyZSB1c2luZyBqYWlscywgdGhleSBh cmUgcHJvYmFibHkgd2hhdCBpcyBjYXVzaW5nDQo+IHRoZSBwcm9ibGVtLiBPbmUga25vd24g amFpbCByZWxhdGVkIGlzc3VlIGlzIHRoYXQgdGhlIG5mc3VzZXJkDQo+IHVwY2FsbHMgdG8g dXNlcmxhbmQgZG9uYHQgd29yayBiZWNhdXNlIHRoZXkgZG9uYHQgY29tZSBmcm9tIDEyNy4w LjAuMS4NCj4gVGhlcmUgYXJlIHByb2JhYmx5IG90aGVycyBJIGFtIG5vdCBhd2FyZSBvZi4N Cg0KIFRoaXMgcGFuaWMgb2NjdXJyZWQgd2hlbiBwb3VkcmllcmUgdXBkYXRlZCBqYWlsIHNv dXJjZXMgZnJvbSBORlMgc2hhcmUuDQpCdXQsIGFzIGZhciBhcyBJIHVuZGVyc3RhbmQsIHNv dXJjZSB1cGRhdGVzIGFyZSBwZXJmb3JtZWQgaW4gaG9zdCBzeXN0ZW0sIGFzDQpqYWlsIGRv ZXNuJ3QgaGF2ZSBORlMgbW91bnRlZC4NCg0KLS0gDQpCZXN0IHJlZ2FyZHMsDQogTGV2ICAg ICAgICAgICAgICAgICAgICAgICAgICAgIG1haWx0bzpsZXZARnJlZUJTRC5vcmc= ------------12219C0062CF59573 Content-Type: application/pgp-signature -----BEGIN PGP MESSAGE----- Version: GnuPG v2.0.22 (MingW32) iQJ8BAEBCgBmBQJWywDaXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EePJtIP/1pD6WMFi9KUk4h7z6w9Ihlk E/o8JisHSZogCaGZDXrkmQN4G3pQaKnyUzHwignt1vwTWXes3/YSTpryIU+l59dP i29t7u4ECpD5F1UzvVoNFxzfSvebdowEcXyWjwCYu4/FAVmjcrIc6xmDSFrFJ3zR oT0NLxTjTxUhlzm911vrpQ5RCvVZaM6Wn9WzpCyzIjm0fqqFBELbpkbySprAVabb 7CfCTvcamF1Kiy+jzkE6xBQ7HDenElM3j/+vL4GhRaJ84p1bNF20RL9ZXBrt4xOc oF2EM43CRjWllxR7/603HhdFUFwC1C5R9cwRHu/h1mQHiJqdlt37/aOuz97nEwKF 1QyBEcwocNiM/rGd8KpHh5MpYoriO8uApXpD+dxfk5ZNuEN7Ff8DzFwqJaTIPEe+ I/R4gXc9EtTj0Uft/aXAhPKlZR8GHZjMktA/eogOWHEqNRg7HkmicJys1tOoijRA w7IyUbRS0wNPMf4RGzwxGpVlwtXmy2d/38APFIIMBYI5L99D7G80MPP8H94jukCk dKk3vgVwK48Co9J71FfnvZZ5N08Yol08NG9/wE8eamQN49U7j1A8p51OE5F8z49M jb3vCu4cI1+nOJV0PHtnd8WCsWFtZ1DhYLr0ek9sRVnG11vFdktHOtXepLOtPUVC m5/2UFJqGMsT837zUVgH =wzf1 -----END PGP MESSAGE----- ------------12219C0062CF59573-- From owner-freebsd-fs@freebsd.org Tue Feb 23 06:05:59 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3AD8AB1CBF for ; Tue, 23 Feb 2016 06:05:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B42301E64 for ; Tue, 23 Feb 2016 06:05:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1N65xAI002230 for ; Tue, 23 Feb 2016 06:05:59 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 06:05:59 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: keywords priority bug_severity cc bug_status flagtypes.name Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 06:05:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |crash, needs-qa, regression Priority|--- |Normal Severity|Affects Only Me |Affects Some People CC| |freebsd-fs@FreeBSD.org, | |re@FreeBSD.org Status|New |Open Flags| |mfc-stable10? --- Comment #1 from Kubilay Kocak --- Thank you for the report Andy If possible, could you: * Confirm whether or not the issue is reproducible on a recent 11.0-CURRENT * Include (as an attachment) another backtrace after the panic if reproduci= ble --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 23 15:46:08 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A2CFAB2261 for ; Tue, 23 Feb 2016 15:46:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A00EBB for ; Tue, 23 Feb 2016 15:46:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1NFk7pd067658 for ; Tue, 23 Feb 2016 15:46:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 15:46:08 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jimharris@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 15:46:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 Jim Harris changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jimharris@FreeBSD.org --- Comment #2 from Jim Harris --- This is a regression due to r293328. This will happen on 11-CURRENT as wel= l. r293328 changed when the controller's ioq array was allocated, such that wh= en we start getting INTx interrupts for the admin queue, ioq is not allocated = yet and caused this panic. See attached patch. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 23 15:46:49 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 191EDAB2318 for ; Tue, 23 Feb 2016 15:46:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0948E1F4 for ; Tue, 23 Feb 2016 15:46:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1NFkmhZ068779 for ; Tue, 23 Feb 2016 15:46:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 15:46:49 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jimharris@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 15:46:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 --- Comment #3 from Jim Harris --- Created attachment 167329 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D167329&action= =3Dedit Patch for bug 207432 --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 23 15:47:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9701CAB23B8 for ; Tue, 23 Feb 2016 15:47:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 87446321 for ; Tue, 23 Feb 2016 15:47:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1NFlIJl069517 for ; Tue, 23 Feb 2016 15:47:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 15:47:19 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jimharris@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: jimharris@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 15:47:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 Jim Harris changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |jimharris@FreeBSD.org --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 23 16:04:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50D21AB2BD1 for ; Tue, 23 Feb 2016 16:04:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4186D11C3 for ; Tue, 23 Feb 2016 16:04:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1NG4IST040291 for ; Tue, 23 Feb 2016 16:04:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 16:04:19 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: wac@google.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: jimharris@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 16:04:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 --- Comment #4 from Andy Carrel --- It's probably worth noting that avoiding INTx with 'hw.pci.honor_msi_blacklist=3D0' in /boot/loader.conf allows things to boot= and function normally. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 23 19:56:43 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E872AB29D4 for ; Tue, 23 Feb 2016 19:56:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FFD9CAC for ; Tue, 23 Feb 2016 19:56:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1NJuhja044908 for ; Tue, 23 Feb 2016 19:56:43 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 19:56:43 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jimharris@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: jimharris@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 19:56:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 --- Comment #5 from Jim Harris --- Hi Andy, Are you able to test the attached patch? I'm pretty sure this fixes your i= ssue but wanted to wait to commit in case you can verify it. Thanks, -Jim --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 23 21:05:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 848C4AB23EB for ; Tue, 23 Feb 2016 21:05:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7750C1365 for ; Tue, 23 Feb 2016 21:05:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1NL5eAr013880 for ; Tue, 23 Feb 2016 21:05:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Tue, 23 Feb 2016 21:05:40 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: wac@google.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: jimharris@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 21:05:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 --- Comment #6 from Andy Carrel --- I can confirm that the attached patch does boot successfully in INTx mode. Thanks! --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Wed Feb 24 00:01:52 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 235BCAB2B8F for ; Wed, 24 Feb 2016 00:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1446A1644 for ; Wed, 24 Feb 2016 00:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1O01pow077423 for ; Wed, 24 Feb 2016 00:01:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207432] panic: nvme_ctrlr_intx_handler Date: Wed, 24 Feb 2016 00:01:52 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: crash, needs-qa, regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: jimharris@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2016 00:01:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207432 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: jimharris Date: Wed Feb 24 00:01:10 UTC 2016 New revision: 295944 URL: https://svnweb.freebsd.org/changeset/base/295944 Log: nvme: fix intx handler to not dereference ioq during initialization This was a regression from r293328, which deferred allocation of the controller's ioq array until after interrupts are enabled during boot. PR: 207432 Reported and tested by: Andy Carrel MFC after: 3 days Sponsored by: Intel Changes: head/sys/dev/nvme/nvme_ctrlr.c --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Thu Feb 25 20:58:12 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00C1FAB413B; Thu, 25 Feb 2016 20:58:12 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-yw0-x22f.google.com (mail-yw0-x22f.google.com [IPv6:2607:f8b0:4002:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA27819CD; Thu, 25 Feb 2016 20:58:11 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-yw0-x22f.google.com with SMTP id e63so53030716ywc.3; Thu, 25 Feb 2016 12:58:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=0fzgag7fJ2WnfKsMMR4hxq2eKaIPPMDqDWzyV7gde3w=; b=0bXXZYjYGbgaKqtaVV4AUwFmeNSgW2JxgNhd1EbRb0SIaWoHO3fGxOf+PDIN+3fAlk hTdHtKktzT5imN0KdoxnwJFAwNxnBItzYae/VCYHzfs1z5HkzqtVUsTP0xsme1G6OWCM dyjAhyvFyrkyMMp754dXLfiN19E2eJHKxDmzOyM0o73NFEBZ6DyL/K8qOlwvfRfrXnyK zbGr3Gi2gXiWKe7/fWm3cogQkB0taTiwEcMNdKwEdCCjKzhXrm/bhZcZAWhJhDCV1s7j /S2cQ/33FgPdk/tNMCdOwHFxwgtsBmDXNgb2lp5Vwq5tStXnS7GAQx8dRqhIYKANuDlf jc6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=0fzgag7fJ2WnfKsMMR4hxq2eKaIPPMDqDWzyV7gde3w=; b=gTkd0vFBIEB04vvNobYuLtGleoUyrUDgfBlnGIm4CpJhj12PapZGKKAD8QT+zYPx5j cDKEDK6HVlU3WsNY0/ShwfZSQ7ayAF6P9+qRyX8dDpCJxPt7p181p+cLtZp1HVL4VU+7 P/4uKWux3gXKQXu4I/5PUXIEt/Tlphb1vWksDM28PEx1NRRDWHjiL194U4Nu7mbsHg7j Rkv+oQUetUAcQHK712qLRAStAIsC7GYwlsWjnvKItI5IwraC3IR4mgHRt8ifz5kCv22Z Gv98h7ZbsENJj0REOP2EAe5cRURKlvXswJQNilnnQ+5A1+9HMZCnL2VTx0Aw1vAqt26p y2Xw== X-Gm-Message-State: AG10YOSldjgOq3cKRl5Y1hRor6JsFyV6/CkdHNP5/T1YInXzs2G1BPplm5FLzYoAHWoyP63jlBGU3SCQIPBeWw== MIME-Version: 1.0 X-Received: by 10.13.232.136 with SMTP id r130mr23950885ywe.2.1456433890980; Thu, 25 Feb 2016 12:58:10 -0800 (PST) Received: by 10.37.27.130 with HTTP; Thu, 25 Feb 2016 12:58:10 -0800 (PST) Date: Thu, 25 Feb 2016 15:58:10 -0500 Message-ID: Subject: ZFS full system backup hoses the backup host. From: Zaphod Beeblebrox To: FreeBSD Hackers , freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2016 20:58:12 -0000 This violated POLA for me. I backed up a host using: time zfs send -vRI @backup-1-e zroot@backup-1-f | ssh backuphost "zfs receive -vFud zroot/backup/host" Only to find that the backup host (a week later) failed to reboot? The problem? Well... -u on receive marks the filesystem as unmounted only "right now" not "next reboot" and -R on send implies -p (send dataset attributes) and... ... zroot/ROOT/default mountpoint=/ (among others). The only hackish way to fix this I see is to have a list of mountpoints to correct --- which is partially what I'm trying to avoid by using -R --- I just want the whole thing backed up. What have other people done to get around this and/or can we either put in an "ignore properties" on receive flag or a -R on send that doesn't send them? From owner-freebsd-fs@freebsd.org Fri Feb 26 04:43:56 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F2C3AB4EE7 for ; Fri, 26 Feb 2016 04:43:56 +0000 (UTC) (envelope-from terehovv@mail.ru) Received: from fallback3.mail.ru (fallback3.mail.ru [94.100.181.189]) by mx1.freebsd.org (Postfix) with ESMTP id 48A2E1859 for ; Fri, 26 Feb 2016 04:43:54 +0000 (UTC) (envelope-from terehovv@mail.ru) Received: from f349.i.mail.ru (f349.i.mail.ru [217.69.140.245]) by fallback3.mail.ru (mPOP.Fallback_MX) with ESMTP id 6B87217023AAB for ; Fri, 26 Feb 2016 07:42:55 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Type:Message-ID:Reply-To:Date:MIME-Version:Subject:To:From; bh=8szJSMsBDNCnOwOpfhAAA8pEkiyU/RexklYYwCfYy84=; b=LEayoxTK/4bNHBfQan3Pq/rlPTfG/o8rYywWB0zRrqXb+LfUhYmj4Xm2Ozng9RAIiUTXmqGJFoDsgopYlUJ0uNFrlxI21O98Bc/BjsL/XpWeqfrwUEkn/nwBZ3Jeow5u+JxZ7WIteeXoOanl9+VbimmSUw+Mzuw/XmXDXjTYa6k=; Received: from [62.231.161.51] (ident=mail) by f349.i.mail.ru with local (envelope-from ) id 1aZAEo-0007LB-SL for freebsd-fs@freebsd.org; Fri, 26 Feb 2016 07:42:47 +0300 Received: from [62.231.161.51] by e.mail.ru with HTTP; Fri, 26 Feb 2016 07:42:46 +0300 From: =?UTF-8?B?0JLRj9GH0LXRgdC70LDQsg==?= To: freebsd-fs@freebsd.org Subject: =?UTF-8?B?VW5hYmxlIHRvIGNyZWF0ZSBaVk9MIChlcnJvcj02KQ==?= MIME-Version: 1.0 X-Mailer: Mail.Ru Mailer 1.0 X-Originating-IP: [62.231.161.51] Date: Fri, 26 Feb 2016 07:42:46 +0300 Reply-To: =?UTF-8?B?0JLRj9GH0LXRgdC70LDQsg==?= X-Priority: 3 (Normal) Message-ID: <1456461766.199245810@f349.i.mail.ru> X-Mras: Ok X-Spam: undefined Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 04:43:56 -0000 IEkgY2hlY2tlZCBjcmVhdGlvbiB6dm9sIHNuYXBzaG90cyBhbmQgSSBoYXZlIHRoaXMgcHJvYmxl bSBpZiB6dm9sIG5hbWUgbGVuZ3RoIHBsdXMgc25hcHNob3QgbmFtZSBsZW5ndGggYXJlIGxvbmdl ciB0aGFuIDU5IHN5bWJvbHMuCgptZGNvbmZpZyAtYSAtdCBtYWxsb2MgLXMgMjAwTQptZDAKenBv b2wgY3JlYXRlIHpmczAgbWQwCnpmcyBzZXQgdm9sbW9kZT1kZXYgemZzMAp6ZnMgY3JlYXRlIHpm czAvYmh5dmVfdm9sdW1lcwp6ZnMgY3JlYXRlIHpmczAvYmh5dmVfdm9sdW1lcy9mcmVlYnNkX2Ft ZDY0XzEwX3N0YWJsZQp6ZnMgc25hcHNob3QgLXIgemZzMC9iaHl2ZV92b2x1bWVzQGAvYmluL2Rh dGUgK1wlSDpcJU06XCVTX1wlZC1cJW0tXCVZYAoKbWRjb25maWcgLWEgLXQgbWFsbG9jIC1zIDIw ME0KbWQxCnpwb29sIGNyZWF0ZSB6ZnMxIG1kMQp6ZnMgY3JlYXRlIHpmczEvYmh5dmVfdm9sdW1l cwp6ZnMgY3JlYXRlIC1zViAxMDBNIC1vIHZvbG1vZGU9ZGV2IHpmczEvYmh5dmVfdm9sdW1lcy9m cmVlYnNkX2FtZDY0XzEwX3N0YWJsZQp6ZnMgc25hcHNob3QgLXIgemZzMS9iaHl2ZV92b2x1bWVz QGAvYmluL2RhdGUgK1wlSDpcJU06XCVTX1wlZC1cJW0tXCVZYAoKemZzIGxpc3QgLXQgc25hcHNo b3QgfCBncmVwIHpmcy4KemZzMC9iaHl2ZV92b2x1bWVzQDIzOjEwOjAwXzIyLTAyLTIwMTbCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCA5S8KgwqDCoMKgwqAgLcKgwqDCoCAxOUvCoCAtCnpmczAvYmh5dmVfdm9sdW1lcy9m cmVlYnNkX2FtZDY0XzEwX3N0YWJsZUAyMzoxMDowMF8yMi0wMi0yMDE2wqDCoMKgwqDCoMKgwqDC oMKgwqAgOUvCoMKgwqDCoMKgIC3CoMKgwqAgMTlLwqAgLQp6ZnMxL2JoeXZlX3ZvbHVtZXNAMjM6 MTE6MzVfMjItMDItMjAxNsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDlLwqDCoMKgwqDCoCAtwqDCoMKgIDE5S8KgIC0K emZzMS9iaHl2ZV92b2x1bWVzL2ZyZWVic2RfYW1kNjRfMTBfc3RhYmxlQDIzOjExOjM1XzIyLTAy LTIwMTbCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIC3CoMKgwqDCoCA4S8KgIC0K CmNhdCAvdmFyL2xvZy9tZXNzYWdlcyB8IGdyZXAgemZzLi8KRmViIDIyIDIzOjExOjM2IGtlcm5l bDogWkZTIFdBUk5JTkc6IFVuYWJsZSB0byBjcmVhdGUgWlZPTCB6ZnMxL2JoeXZlX3ZvbHVtZXMv ZnJlZWJzZF9hbWQ2NF8xMF9zdGFibGVAMjM6MTE6MzVfMjItMDItMjAxNiAoZXJyb3I9NikuCgp6 ZnMgZ2V0IC1yIHR5cGUgemZzMApOQU1FwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBQUk9QRVJUWcKgIFZBTFVFwqDCoMKgwqDCoMKg IFNPVVJDRQp6ZnMwwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCB0eXBlwqDCoMKgwqDCoCBmaWxlc3lzdGVtwqAgLQp6ZnMwL2JoeXZl X3ZvbHVtZXPCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgdHlwZcKgwqDCoMKgwqAg ZmlsZXN5c3RlbcKgIC0KemZzMC9iaHl2ZV92b2x1bWVzQDIzOjEwOjAwXzIyLTAyLTIwMTbCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCB0eXBlwqDCoMKg wqDCoCBzbmFwc2hvdMKgwqDCoCAtCnpmczAvYmh5dmVfdm9sdW1lcy9mcmVlYnNkX2FtZDY0XzEw X3N0YWJsZcKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCB0eXBlwqDC oMKgwqDCoCBmaWxlc3lzdGVtwqAgLQp6ZnMwL2JoeXZlX3ZvbHVtZXMvZnJlZWJzZF9hbWQ2NF8x MF9zdGFibGVAMjM6MTA6MDBfMjItMDItMjAxNsKgIHR5cGXCoMKgwqDCoMKgIHNuYXBzaG90wqDC oMKgIC0KCnpmcyBnZXQgLXIgdHlwZSB6ZnMxCk5BTUXCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIFBST1BFUlRZwqAgVkFMVUXCoMKg wqDCoMKgwqAgU09VUkNFCnpmczHCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHR5cGXCoMKgwqDCoMKgIGZpbGVzeXN0ZW3CoCAtCnpm czEvYmh5dmVfdm9sdW1lc8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCB0eXBlwqDC oMKgwqDCoCBmaWxlc3lzdGVtwqAgLQp6ZnMxL2JoeXZlX3ZvbHVtZXNAMjM6MTE6MzVfMjItMDIt MjAxNsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHR5 cGXCoMKgwqDCoMKgIHNuYXBzaG90wqDCoMKgIC0KemZzMS9iaHl2ZV92b2x1bWVzL2ZyZWVic2Rf YW1kNjRfMTBfc3RhYmxlwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg IHR5cGXCoMKgwqDCoMKgIHZvbHVtZcKgwqDCoMKgwqAgLQp6ZnMxL2JoeXZlX3ZvbHVtZXMvZnJl ZWJzZF9hbWQ2NF8xMF9zdGFibGVAMjM6MTE6MzVfMjItMDItMjAxNsKgIHR5cGXCoMKgwqDCoMKg IHNuYXBzaG90wqDCoMKgIC0KCg== From owner-freebsd-fs@freebsd.org Fri Feb 26 05:11:34 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB804AB56AB; Fri, 26 Feb 2016 05:11:34 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from exch2-4.slu.se (pop.slu.se [77.235.224.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 434D51A6; Fri, 26 Feb 2016 05:11:33 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from Exch2-3.slu.se (77.235.224.123) by exch2-4.slu.se (77.235.224.124) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Fri, 26 Feb 2016 05:56:19 +0100 Received: from Exch2-3.slu.se ([fe80::e8c9:5585:730f:f95]) by Exch2-3.slu.se ([fe80::e8c9:5585:730f:f95%23]) with mapi id 15.00.1156.000; Fri, 26 Feb 2016 05:56:19 +0100 From: =?iso-8859-1?Q?Karli_Sj=F6berg?= To: Zaphod Beeblebrox CC: FreeBSD Hackers , freebsd-fs Subject: Re: ZFS full system backup hoses the backup host. Thread-Topic: ZFS full system backup hoses the backup host. Thread-Index: AQHRcA9G+6fLnprz0kSutt/jUfgKRZ89w/HY Date: Fri, 26 Feb 2016 04:56:19 +0000 Message-ID: <0ae0e3ab-6ad1-4ae2-9ee9-a7d993088a01@email.android.com> References: In-Reply-To: Accept-Language: sv-SE, en-US Content-Language: sv-SE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 05:11:34 -0000 Den 25 feb. 2016 9:58 em skrev Zaphod Beeblebrox : > > This violated POLA for me. I backed up a host u This violated POLA for me. I backed up a host using: time zfs send -vRI @backup-1-e zroot@backup-1-f | ssh backuphost "zfs receive -vFud zroot/backup/host" Only to find that the backup host (a week later) failed to reboot? The problem? Well... -u on receive marks the filesystem as unmounted only "right now" not "next reboot" and -R on send implies -p (send dataset attributes) and... ... zroot/ROOT/default mountpoint=3D/ (among others). The only hackish way to fix this I see is to have a list of mountpoints to correct --- which is partially what I'm trying to avoid by using -R --- I just want the whole thing backed up. What have other people done to get around this and/or can we either put in an "ignore properties" on receive flag or a -R on send that doesn't send them? _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Fri Feb 26 07:07:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35BDBAB405F for ; Fri, 26 Feb 2016 07:07:03 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 26587108B for ; Fri, 26 Feb 2016 07:07:03 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 22AFBAB405E; Fri, 26 Feb 2016 07:07:03 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22394AB405D for ; Fri, 26 Feb 2016 07:07:03 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail107.syd.optusnet.com.au (mail107.syd.optusnet.com.au [211.29.132.53]) by mx1.freebsd.org (Postfix) with ESMTP id C6C73108A for ; Fri, 26 Feb 2016 07:07:02 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail107.syd.optusnet.com.au (Postfix) with ESMTPS id D03C0D49C1B for ; Fri, 26 Feb 2016 18:06:54 +1100 (AEDT) Date: Fri, 26 Feb 2016 18:06:53 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: fs@freebsd.org Subject: silly write caching in nfs3 Message-ID: <20160226164613.N2180@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=EfU1O6SC c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=Y_XGasR6flW_SNLsOy4A:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 07:07:03 -0000 nfs3 is slower than in old versions of FreeBSD. I debugged one of the reasons today. Writes have apparently always done silly caching. Typical behaviour is for iozone writing a 512MB file where the file fits in the buffer cache/VMIO. The write is cached perfectly. But then when nfs_open() reeopens the file, it calls vinvalbuf() to discard all of the cached data. Thus nfs write caching usually discards useful older data to make space for newer data that will never be never used (unless the file is opened r/w and read using the same fd (and is not accessed for a setattr or advlock operation -- these call vinvalbuf() too, if NMODIFIED)). The discarding may be delayed for a long time. Then keeping the useless data causes even more older data to be discarded. Discarding it on close would at least prevent further loss. It would have to be committed on close before discarding it of course. Committing it on close does some good things even without discarding there, and in oldnfs it gives a bug that prevents discaring in open -- see below. nfs_open() does the discarding for different reasons in the NMODIFIED and !NMODIFIED cases. In the NMODIFED case, it discard unconditionally. This case can be avoided by fsync() before close or setting the sysctl to commit in close. iozone does he fsync(). This helps in oldnfs but not in newfs. With it, iozone on newfs now behaves like it did on oldnfs 10-20 years ago. Something (perhaps just the timestamp bugs discussed later) "fixed" the discarding on oldnfs 5-10 years ago. I think not committing in close is supposed to be an optimization, but it is actually a pessimization for my kernel build tests (with object files on nfs, which I normally avoid). Builds certainly have to reopen files after writing them, to link them and perhaps to install them. This causes the discarding. My kernel build tests also do a lot of utimes() calls which cause the discarding before commit-on-close can avoid the above cause for it it by clearing NMODIFIED. Enabling commit-on-close gives a small optimisation with oldnfs by avoiding all of the discarding except for utimes(). It reduces read RPCs by about 25% without increasing write RPCs or real time. It decreases real time by a few percent. The other reason for discarding is because the timestamps changed -- you just wrote them, so the timestamps should have changed. Different bugs in comparing the timestamps gave different misbehaviours. In old versions of FreeBSD and/or nfs, the timestamps had seconds granularity, so many changes were missed. This explains mysterious behaviours by iozone 10-20 years ago: the write caching is seen to work perfectly for most small total sizes, since all the writes take less than 1 second so the timestamps usually don't change (but sometimes the writes lie across a seconds boundary so the timestamps do change). oldnfs was fixed many years ago to use timestamps with nanoseconds resolution, but it doesn't suffer from the discarding in nfs_open() in the !NMODIFIED case which is reached by either fsync() before close of commit on close. I think this is because it updates n_mtime to the server's new timestamp in nfs_writerpc(). This seems to be wrong, since the file might have been written to by other clients and then the change would not be noticed until much later if ever (setting the timestamp prevents seeing it change when it is checked later, but you might be able to see another metadata change). newfs has quite different code for nfs_writerpc(). Most of it was moved to another function in nanother file. I understand this even less, but it doesn't seem to have fetch the server's new timestamp or update n_mtime in the v3 case. There are many other reasons why nfs is slower than in old versions. One is that writes are more often done out of order. This tends to give a slowness factor of about 2 unless the server can fix up the order. I use an old server which can do the fixup for old clients but not for newer clients starting in about FreeBSD-9 (or 7?). I suspect that this is just because Giant locking in old clients gave accidental serialization. Multiple nfsiod's and/or nfsd's are are clearly needed for performance if you have multiple NICs serving multiple mounts. Other cases are less clear. For the iozone benchmark, there is only 1 stream and multiple nfsiod's pessimize it into multiple streams that give buffers which arrive out of order on the server if the multiple nfsiod's are actually active. I use the following configuration to ameliorate this, but the slowness factor is still often about 2 for iozone: - limit nfsd's to 4 - limit nfsiod's to 4 - limit nfs i/o sizes to 8K. The server fs block size is 16K, and using a smaller block size usually helps by giving some delayed writes which can be clustered better. (The non-nfs parts of the server could be smarter and do this intentionally. The out-of-order buffers look like random writes to the server.) 16K i/o sizes otherwise work OK, but 32K i/o sizes are much slower for unknown reasons. Bruce From owner-freebsd-fs@freebsd.org Fri Feb 26 09:48:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02FD7AB5B4D for ; Fri, 26 Feb 2016 09:48:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E876BE27 for ; Fri, 26 Feb 2016 09:48:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1Q9mf9q085706 for ; Fri, 26 Feb 2016 09:48:41 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207464] Panic when destroying ZFS snapshot on boot filesystem Date: Fri, 26 Feb 2016 09:48:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: rep_platform assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 09:48:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207464 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Hardware|amd64 |Any Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Feb 26 11:16:22 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A2D1AB41BA for ; Fri, 26 Feb 2016 11:16:22 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu1176c.smtpx.saremail.com (cu1176c.smtpx.saremail.com [195.16.148.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0C8861DCE for ; Fri, 26 Feb 2016 11:16:21 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPSA id 601B59DC95A; Fri, 26 Feb 2016 12:07:18 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: ZFS receive and attributes, proposing some changes. was Re: ZFS full system backup hoses the backup host. From: Borja Marcos In-Reply-To: <0ae0e3ab-6ad1-4ae2-9ee9-a7d993088a01@email.android.com> Date: Fri, 26 Feb 2016 12:07:17 +0100 Cc: Zaphod Beeblebrox , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: References: <0ae0e3ab-6ad1-4ae2-9ee9-a7d993088a01@email.android.com> To: =?utf-8?Q?Karli_Sj=C3=B6berg?= X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 11:16:22 -0000 > On 26 Feb 2016, at 05:56, Karli Sj=C3=B6berg = wrote: > What have other people done to get around this and/or can we either = put in > an "ignore properties" on receive flag or a -R on send that doesn't = send > them? (I removed freebsd-hackers as this discussion belongs in freebsd-fs) The possibility of replacing options in the receive command would be = most useful here. Ideally it should be done atomically with the receive.=20 For example: Imagine a replication system. It would be something like = this: - Make the first snapshot - Send it | receive it with -u (you don=E2=80=99t want it mounted) - Set the destination dataset=E2=80=99s canmount property to =E2=80=9Cnoau= to" - And, periodically - Make new snapshot - Send it incrementally - etc etc However, there is a race condition here. If your replication = program/script crashes, or it stops for whatever reason between the first send and the =E2=80=9Cset canmount=E2=80= =9D (or you reboot the destination server) you end up with a time bomb. Despite being received with the -u flag, if = you boot it will be mounted automatically. If you are replicating the root filesystem (or anything = mounted on an important system directory) you are dead :) To avoid this kind of problems, two options come to my mind, from least = desirable/useful/consistent to better (in my opinion) 1) Have some kind of force-inherit attribute on a dataset which forces = children=E2=80=99s mountpoint to go below it. Example: when replicating server1=E2=80=99s datasets, I send = them below to pool/server1_copies. If I set mountpoint=3Dforce_child (or something similar) for = pool/server1_copies, anything received under it would have the mountpoint changed, adding the mountpoint of = pool/server1_copies to it. Imagine I am replicating the /usr filesystem of server1, which = is pool_server1/usr, with the mountpoint /usr. The destination would have the mountpoint = /pool/server1_copies/usr. This at least avoids the unintended mountpoint overwrite = problem, which is not bad. And this possibility can be useful, but it=E2=80=99s not a very clean solution. 2) Adding an option to =E2=80=9Czfs receive=E2=80=9D so that properties = can be changed for the received dataset *atomically*. The race condition is avoided if I can specify an option to zfs = recv, something like -O canmount=3Dnoauto.=20 Actually, there are some changes I would make, not just for this = particular case, which will not break functionality and will make ZFS safer to use and easier to handle, = especially when replicating datasets. 1) Enhancing =E2=80=9Czfs recv=E2=80=9D with the possibility of changing = options and holds atomically.=20 - Changing options: -O option=3Dvalue -O option=3Dvalue - Adding holds: -h holdname When using snapshots to replicate datasets you must be careful = not to delete the last replicated snapshot accidentally, or you can lose the ability to do an incremental = send, forcing a full one. Holds help to prevent this. And if using them it would be very useful to have the last = snapshot properly protected in the destination dataset as well. Again, it should be done ATOMICALLY. Same applies to dataset properties, if only for the outright = dangerous combination of the canmount/mountpoint properties. Once more, we want this to be done ATOMICALLY with = the zfs recv.=20 2) Adding the possibility of specifying =E2=80=9Cdefault=E2=80=9D or = =E2=80=9Cinherit=E2=80=9D for the mountpoint property. When a dataset is created, the default mountpoint is = pool/dataset_name unless a different value is specified. A value of =E2=80=9Cdefault=E2=80=9D would set the dataset=E2=80=99= s mountpoint back to pool/=E2=80=A6/dataset_name, and =E2=80=9Cinherit=E2=80= =9D could=20 set the mountpoint to the "parent=E2=80=99s = mountpoint=E2=80=9D/dataset_name. These two options can be very useful when using zfs send/zfs = recv to keep replicas as a backup, but I think they can be useful in other situations.=20 What do you think? In my opinion these additions won=E2=80=99t break = functionality and they are worthwhile if only for making replication safer/more useful. Best regards, Borja. =09 From owner-freebsd-fs@freebsd.org Fri Feb 26 12:11:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BE60AB5F5F for ; Fri, 26 Feb 2016 12:11:09 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8600E2BA for ; Fri, 26 Feb 2016 12:11:08 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22f.google.com with SMTP id c200so69640040wme.0 for ; Fri, 26 Feb 2016 04:11:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=BAHfQc86pr00hc9eMElrTa02qlJ75nBpNDDPbN5iwA8=; b=ygrWE4jq2AyHmrmauQr2yDZK+fSLrfuagihZXgCeOiJOt5Oc4VMCuqgTeAvsFPEir0 vJSEHt/CeF6ejYZG2OTPEVkMStk1bbYyU5JT44uspzNLJcK3aWDFC3tYx2TEMAxR9jOl v9aZGf/HpzZkXz6LgniMnxrYkACw13D44vKdTflOTW7dGx9Hw4hmJp7bW/ey1k7opMYw QUuwW9FwLqFIzY+g/Vl1oX3yFtLwE/CY4Ogq8VUbs7ryFV+pvs+fZG35VaCF78bECQVT OyVKl4nTZX4X2XWm9Smk3DZ4UiycI0T5ZT9RAPZ8WLJfzUkC2Yt1XSorZoQxXzxN+71r rVag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=BAHfQc86pr00hc9eMElrTa02qlJ75nBpNDDPbN5iwA8=; b=RbZSJeOUxwyRjmF0BZ/FMMDyjAGXkyyJl8O764r0BtkHB9PU7NWRu+sswHkZi2xyRQ Iv/vL13t5C3KzAszU76NIfTpgUI8+eswoE8CFzr9C9PVskbUZ6GnUuAae+lRNdizTvW3 e+yUE2VTlXqWvUTJvSpZSgW18GAvPgKm1houRSZ2s4ekFBpvjGm62mCMQ8Qz3vY+haIb ql6IuwTkn7Yt3+ByTdB8a7yxETpRUnEL/RPlJrc9TwYW6Cs8zzdfPCWYqCWruirlsrfq MZAsInb8VVDOcyR6gwO8/cceBjt7lgjsffZl+03POSk7NdHBSmFy3yI9Uxcb88E+CabJ ZIJw== X-Gm-Message-State: AD7BkJJGHVaso2Lh6kcFv3hjeLkL50OGcc3zlQPhOwO8TuvEQj+WCfVoio185VXBxL2WgAKC X-Received: by 10.194.184.112 with SMTP id et16mr1359007wjc.75.1456488666445; Fri, 26 Feb 2016 04:11:06 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id v22sm2632243wmv.12.2016.02.26.04.11.05 for (version=TLSv1/SSLv3 cipher=OTHER); Fri, 26 Feb 2016 04:11:05 -0800 (PST) Subject: Re: ZFS receive and attributes, proposing some changes. was Re: ZFS full system backup hoses the backup host. To: freebsd-fs@freebsd.org References: <0ae0e3ab-6ad1-4ae2-9ee9-a7d993088a01@email.android.com> From: Steven Hartland Message-ID: <56D040E1.3090000@multiplay.co.uk> Date: Fri, 26 Feb 2016 12:11:13 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/mixed; boundary="------------060909070708000505060205" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 12:11:09 -0000 This is a multi-part message in MIME format. --------------060909070708000505060205 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit I actually have a patch (see attached) which does this but never got round to finishing the RTI so its got some rough edges. On 26/02/2016 11:07, Borja Marcos wrote: >> On 26 Feb 2016, at 05:56, Karli Sjöberg wrote: >> What have other people done to get around this and/or can we either put in >> an "ignore properties" on receive flag or a -R on send that doesn't send >> them? > (I removed freebsd-hackers as this discussion belongs in freebsd-fs) > > > The possibility of replacing options in the receive command would be most useful here. Ideally > it should be done atomically with the receive. > > For example: Imagine a replication system. It would be something like this: > > - Make the first snapshot > > - Send it | receive it with -u (you don’t want it mounted) > > - Set the destination dataset’s canmount property to “noauto" > > - And, periodically > > - Make new snapshot > > - Send it incrementally > > - etc etc > > > However, there is a race condition here. If your replication program/script crashes, or it stops for > whatever reason between the first send and the “set canmount” (or you reboot the destination server) > you end up with a time bomb. Despite being received with the -u flag, if you boot it will be mounted > automatically. If you are replicating the root filesystem (or anything mounted on an important system > directory) you are dead :) > > To avoid this kind of problems, two options come to my mind, from least desirable/useful/consistent > to better (in my opinion) > > 1) Have some kind of force-inherit attribute on a dataset which forces children’s mountpoint to go below it. > > Example: when replicating server1’s datasets, I send them below to pool/server1_copies. If I set > mountpoint=force_child (or something similar) for pool/server1_copies, anything received under it > would have the mountpoint changed, adding the mountpoint of pool/server1_copies to it. > > Imagine I am replicating the /usr filesystem of server1, which is pool_server1/usr, with the > mountpoint /usr. The destination would have the mountpoint /pool/server1_copies/usr. > > This at least avoids the unintended mountpoint overwrite problem, which is not bad. And this possibility > can be useful, but it’s not a very clean solution. > > 2) Adding an option to “zfs receive” so that properties can be changed for the received dataset *atomically*. > > The race condition is avoided if I can specify an option to zfs recv, something like -O canmount=noauto. > > > Actually, there are some changes I would make, not just for this particular case, which will not break > functionality and will make ZFS safer to use and easier to handle, especially when replicating datasets. > > > 1) Enhancing “zfs recv” with the possibility of changing options and holds atomically. > > - Changing options: -O option=value -O option=value > - Adding holds: -h holdname > > When using snapshots to replicate datasets you must be careful not to delete the last replicated snapshot > accidentally, or you can lose the ability to do an incremental send, forcing a full one. Holds help to prevent this. > And if using them it would be very useful to have the last snapshot properly protected in the destination dataset > as well. Again, it should be done ATOMICALLY. > > Same applies to dataset properties, if only for the outright dangerous combination of the canmount/mountpoint > properties. Once more, we want this to be done ATOMICALLY with the zfs recv. > > 2) Adding the possibility of specifying “default” or “inherit” for the mountpoint property. > > When a dataset is created, the default mountpoint is pool/dataset_name unless a different value is specified. > A value of “default” would set the dataset’s mountpoint back to pool/…/dataset_name, and “inherit” could > set the mountpoint to the "parent’s mountpoint”/dataset_name. > > These two options can be very useful when using zfs send/zfs recv to keep replicas as a backup, but I think > they can be useful in other situations. > > > What do you think? In my opinion these additions won’t break functionality and they are worthwhile if only for making > replication safer/more useful. > > Best regards, > > > > > > > > Borja. > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --------------060909070708000505060205 Content-Type: text/plain; charset=UTF-8; name="zfs-recv-properties.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="zfs-recv-properties.patch" QWRkcyBzdXBwb3J0IGZvciBwcm9wZXJ0eSBvdmVycmlkZXMgKC1vIHByb3BlcnR5PXZhbHVl KSwgcHJvcGVydHkgZXhjbHVkZXMKKC14IHByb3BlcnR5KSBhbmQgZGF0YXNldCBsaW1pdHMg KC1sIDx2b2x1bWV8ZmlsZXN5c3RlbT4pIHRvIHpmcyByZWNlaXZlLgoKQm90aCAtbyBhbmQg LXggb3B0aW9ucyBtaXJyb3IgdGhlIGZ1bmN0aW9uYWxpdHkgYWxyZWFkeSBhdmFpbGFibGUg aW4KT3JhY2xlJ3MgWkZTIGltcGxlbWVudGF0aW9uIHdoaWNoIGlzIGFsc28gbWVudGlvbmVk IGluIHRoZSB1cHN0cmVhbQpmZWF0dXJlIHJlcXVlc3QgIzI3NDU6Cmh0dHBzOi8vd3d3Lmls bHVtb3Mub3JnL2lzc3Vlcy8yNzQ1CgpUaGUgLWwgb3B0aW9uIGFsbG93cyByZWNlaXZlIHRv IGJlIGxpbWl0ZWQgdG8gc3BlY2lmaWMgZGF0YXNldHMgd2l0aGluCnRoZSBzdHJlYW0gZWZm ZWN0aXZlbHkgYWxsb3dpbmcgcGFydGlhbCByZXN0b3JlcyBmcm9tIGEgbXVsdGkgZGF0YXNl dApzdHJlYW0uCi0tLSBjZGRsL2NvbnRyaWIvb3BlbnNvbGFyaXMvbGliL2xpYnpmcy9jb21t b24vbGliemZzX2RhdGFzZXQuYy5vcmlnCTIwMTQtMTAtMTAgMDk6MzE6MzYuMDAwMDAwMDAw ICswMDAwCisrKyBjZGRsL2NvbnRyaWIvb3BlbnNvbGFyaXMvbGliL2xpYnpmcy9jb21tb24v bGliemZzX2RhdGFzZXQuYwkyMDE0LTExLTEzIDE0OjMxOjEwLjQ1MjY3MDEzMSArMDAwMApA QCAtMjg3NCw3ICsyODc0LDcgQEAgcGFyZW50X25hbWUoY29uc3QgY2hhciAqcGF0aCwgY2hh ciAqYnVmLAogICogJ3pvbmVkJyBwcm9wZXJ0eSwgd2hpY2ggaXMgdXNlZCB0byB2YWxpZGF0 ZSBwcm9wZXJ0eSBzZXR0aW5ncyB3aGVuIGNyZWF0aW5nCiAgKiBuZXcgZGF0YXNldHMuCiAg Ki8KLXN0YXRpYyBpbnQKK2ludAogY2hlY2tfcGFyZW50cyhsaWJ6ZnNfaGFuZGxlX3QgKmhk bCwgY29uc3QgY2hhciAqcGF0aCwgdWludDY0X3QgKnpvbmVkLAogICAgIGJvb2xlYW5fdCBh Y2NlcHRfYW5jZXN0b3IsIGludCAqcHJlZml4bGVuKQogewotLS0gY2RkbC9jb250cmliL29w ZW5zb2xhcmlzL2xpYi9saWJ6ZnMvY29tbW9uL2xpYnpmc19pbXBsLmgub3JpZwkyMDE0LTEw LTEwIDA5OjMxOjM2LjAwMDAwMDAwMCArMDAwMAorKysgY2RkbC9jb250cmliL29wZW5zb2xh cmlzL2xpYi9saWJ6ZnMvY29tbW9uL2xpYnpmc19pbXBsLmgJMjAxNC0xMS0xMyAxNDozMTox MC40NTM2Njk4MDYgKzAwMDAKQEAgLTE4OCw2ICsxODgsOCBAQCBpbnQgY2hhbmdlbGlzdF9o YXN6b25lZGNoaWxkKHByb3BfY2hhbmdlCiAKIHZvaWQgcmVtb3ZlX21vdW50cG9pbnQoemZz X2hhbmRsZV90ICopOwogaW50IGNyZWF0ZV9wYXJlbnRzKGxpYnpmc19oYW5kbGVfdCAqLCBj aGFyICosIGludCk7CitpbnQgY2hlY2tfcGFyZW50cyhsaWJ6ZnNfaGFuZGxlX3QgKmhkbCwg Y29uc3QgY2hhciAqcGF0aCwgdWludDY0X3QgKnpvbmVkLAorICAgIGJvb2xlYW5fdCBhY2Nl cHRfYW5jZXN0b3IsIGludCAqcHJlZml4bGVuKTsKIGJvb2xlYW5fdCBpc2FfY2hpbGRfb2Yo Y29uc3QgY2hhciAqZGF0YXNldCwgY29uc3QgY2hhciAqcGFyZW50KTsKIAogemZzX2hhbmRs ZV90ICptYWtlX2RhdGFzZXRfaGFuZGxlKGxpYnpmc19oYW5kbGVfdCAqLCBjb25zdCBjaGFy ICopOwotLS0gY2RkbC9jb250cmliL29wZW5zb2xhcmlzL2xpYi9saWJ6ZnMvY29tbW9uL2xp Ynpmc19zZW5kcmVjdi5jLm9yaWcJMjAxNC0xMC0xMCAwOTozMTozNi4wMDAwMDAwMDAgKzAw MDAKKysrIGNkZGwvY29udHJpYi9vcGVuc29sYXJpcy9saWIvbGliemZzL2NvbW1vbi9saWJ6 ZnNfc2VuZHJlY3YuYwkyMDE0LTExLTEzIDE0OjMxOjEwLjQ1NjY2OTQwOCArMDAwMApAQCAt NjUsNyArNjUsOCBAQCBleHRlcm4gdm9pZCB6ZnNfc2V0cHJvcF9lcnJvcihsaWJ6ZnNfaGFu CiAjZGVmaW5lCUVOT0RBVEEJRUlEUk0KIAogc3RhdGljIGludCB6ZnNfcmVjZWl2ZV9pbXBs KGxpYnpmc19oYW5kbGVfdCAqLCBjb25zdCBjaGFyICosIHJlY3ZmbGFnc190ICosCi0gICAg aW50LCBjb25zdCBjaGFyICosIG52bGlzdF90ICosIGF2bF90cmVlX3QgKiwgY2hhciAqKiwg aW50LCB1aW50NjRfdCAqKTsKKyAgICBpbnQsIG52bGlzdF90ICosIG52bGlzdF90ICosIGNv bnN0IGNoYXIgKiwgbnZsaXN0X3QgKiwgYXZsX3RyZWVfdCAqLAorICAgIGNoYXIgKiosIGlu dCwgdWludDY0X3QgKik7CiAKIHN0YXRpYyBjb25zdCB6aW9fY2tzdW1fdCB6ZXJvX2Nrc3Vt ID0geyAwIH07CiAKQEAgLTIwMjQsNyArMjAyNSw3IEBAIGNyZWF0ZWRfYmVmb3JlKGxpYnpm c19oYW5kbGVfdCAqaGRsLCBhdmwKIHN0YXRpYyBpbnQKIHJlY3ZfaW5jcmVtZW50YWxfcmVw bGljYXRpb24obGliemZzX2hhbmRsZV90ICpoZGwsIGNvbnN0IGNoYXIgKnRvZnMsCiAgICAg cmVjdmZsYWdzX3QgKmZsYWdzLCBudmxpc3RfdCAqc3RyZWFtX252LCBhdmxfdHJlZV90ICpz dHJlYW1fYXZsLAotICAgIG52bGlzdF90ICpyZW5hbWVkKQorICAgIG52bGlzdF90ICpyZW5h bWVkLCBudmxpc3RfdCAqbGltaXRkcykKIHsKIAludmxpc3RfdCAqbG9jYWxfbnYsICpkZWxl dGVkID0gTlVMTDsKIAlhdmxfdHJlZV90ICpsb2NhbF9hdmw7CkBAIC0yMDc2LDYgKzIwNzcs MTIgQEAgYWdhaW46CiAJCSAgICAmcGFyZW50X2Zyb21zbmFwX2d1aWQpKTsKIAkJKHZvaWQp IG52bGlzdF9sb29rdXBfdWludDY0KG52ZnMsICJvcmlnaW4iLCAmb3JpZ2luZ3VpZCk7CiAK KwkJaWYgKCFudmxpc3RfZW1wdHkobGltaXRkcykgJiYgIW52bGlzdF9leGlzdHMobGltaXRk cywgZnNuYW1lKSkgeworCQkJaWYgKGZsYWdzLT52ZXJib3NlKQorCQkJCSh2b2lkKSBwcmlu dGYoInNraXBwaW5nIHJlcGxpY2F0aW9uIG9mICVzXG4iLCBmc25hbWUpOworCQkJY29udGlu dWU7CisJCX0KKwogCQkvKgogCQkgKiBGaXJzdCBmaW5kIHRoZSBzdHJlYW0ncyBmcywgc28g d2UgY2FuIGNoZWNrIGZvcgogCQkgKiBhIGRpZmZlcmVudCBvcmlnaW4gKGR1ZSB0byAiemZz IHByb21vdGUiKQpAQCAtMjMzMiw4ICsyMzM5LDkgQEAgZG9hZ2FpbjoKIAogc3RhdGljIGlu dAogemZzX3JlY2VpdmVfcGFja2FnZShsaWJ6ZnNfaGFuZGxlX3QgKmhkbCwgaW50IGZkLCBj b25zdCBjaGFyICpkZXN0bmFtZSwKLSAgICByZWN2ZmxhZ3NfdCAqZmxhZ3MsIGRtdV9yZXBs YXlfcmVjb3JkX3QgKmRyciwgemlvX2Nrc3VtX3QgKnpjLAotICAgIGNoYXIgKip0b3BfemZz LCBpbnQgY2xlYW51cF9mZCwgdWludDY0X3QgKmFjdGlvbl9oYW5kbGVwKQorICAgIHJlY3Zm bGFnc190ICpmbGFncywgbnZsaXN0X3QgKmV4cHJvcHMsIG52bGlzdF90ICpsaW1pdGRzLAor ICAgIGRtdV9yZXBsYXlfcmVjb3JkX3QgKmRyciwgemlvX2Nrc3VtX3QgKnpjLCBjaGFyICoq dG9wX3pmcywgaW50IGNsZWFudXBfZmQsCisgICAgdWludDY0X3QgKmFjdGlvbl9oYW5kbGVw KQogewogCW52bGlzdF90ICpzdHJlYW1fbnYgPSBOVUxMOwogCWF2bF90cmVlX3QgKnN0cmVh bV9hdmwgPSBOVUxMOwpAQCAtMjQ1Myw3ICsyNDYxLDcgQEAgemZzX3JlY2VpdmVfcGFja2Fn ZShsaWJ6ZnNfaGFuZGxlX3QgKmhkbAogCQkJfQogCiAJCQlzb2Z0ZXJyID0gcmVjdl9pbmNy ZW1lbnRhbF9yZXBsaWNhdGlvbihoZGwsIHRvZnMsIGZsYWdzLAotCQkJICAgIHN0cmVhbV9u diwgc3RyZWFtX2F2bCwgcmVuYW1lZCk7CisJCQkgICAgc3RyZWFtX252LCBzdHJlYW1fYXZs LCByZW5hbWVkLCBsaW1pdGRzKTsKIAogCQkJLyogVW5tb3VudCByZW5hbWVkIGZpbGVzeXN0 ZW1zIGJlZm9yZSByZWNlaXZpbmcuICovCiAJCQl3aGlsZSAoKHBhaXIgPSBudmxpc3RfbmV4 dF9udnBhaXIocmVuYW1lZCwKQEAgLTI0OTksOCArMjUwNyw4IEBAIHpmc19yZWNlaXZlX3Bh Y2thZ2UobGliemZzX2hhbmRsZV90ICpoZGwKIAkJICogcmVjdl9za2lwKCkgYW5kIHJldHVy biAwKS4KIAkJICovCiAJCWVycm9yID0gemZzX3JlY2VpdmVfaW1wbChoZGwsIGRlc3RuYW1l LCBmbGFncywgZmQsCi0JCSAgICBzZW5kZnMsIHN0cmVhbV9udiwgc3RyZWFtX2F2bCwgdG9w X3pmcywgY2xlYW51cF9mZCwKLQkJICAgIGFjdGlvbl9oYW5kbGVwKTsKKwkJICAgIGV4cHJv cHMsIGxpbWl0ZHMsIHNlbmRmcywgc3RyZWFtX252LCBzdHJlYW1fYXZsLCB0b3BfemZzLAor CQkgICAgY2xlYW51cF9mZCwgYWN0aW9uX2hhbmRsZXApOwogCQlpZiAoZXJyb3IgPT0gRU5P REFUQSkgewogCQkJZXJyb3IgPSAwOwogCQkJYnJlYWs7CkBAIC0yNTE0LDcgKzI1MjIsNyBA QCB6ZnNfcmVjZWl2ZV9wYWNrYWdlKGxpYnpmc19oYW5kbGVfdCAqaGRsCiAJCSAqIHJlbmFt ZXMgYWdhaW4uCiAJCSAqLwogCQlzb2Z0ZXJyID0gcmVjdl9pbmNyZW1lbnRhbF9yZXBsaWNh dGlvbihoZGwsIHRvZnMsIGZsYWdzLAotCQkgICAgc3RyZWFtX252LCBzdHJlYW1fYXZsLCBO VUxMKTsKKwkJICAgIHN0cmVhbV9udiwgc3RyZWFtX2F2bCwgTlVMTCwgbGltaXRkcyk7CiAJ fQogCiBvdXQ6CkBAIC0yNjI3LDMwICsyNjM1LDE5OCBAQCByZWN2X3NraXAobGliemZzX2hh bmRsZV90ICpoZGwsIGludCBmZCwgCiB9CiAKIC8qCisgKiBDYWxjdWxhdGUgYSBsaXN0IG9m IHByb3BlcnRpZXMgZm9yIHRoZSBjdXJyZW50IGRhdGFzZXQgdGFraW5nIGludG8gYWNjb3Vu dAorICogaXRzIGN1cnJlbnQgcHJvcGVydGllcyAocHJvcHMpIGFuZCB0aGUgZXh0ZXJuYWwg cHJvcGVydGllcyAoZXhwcm9wcykKKyAqCisgKiBUaGlzIGNhbGN1bGF0aW9uOgorICogLSBS ZW1vdmVzIGV4Y2x1ZGVkIHByb3BlcnRpZXMgKGJvb2xlYW5zKQorICogLSBDaGFuZ2VzIHRo ZSB2YWx1ZXMgb2Ygb3ZlcnJpZGVuIHByb3BlcnRpZXMgKHN0cmluZ3MpCisgKgorICogVGhl cmUgYXJlIHR3byB0eXBlcyBvZiBleHRlcm5hbCBwcm9wZXJ0aWVzOgorICogLSBHbG9iYWwg cHJvcGVydGllcworICogLSBEYXRhc2V0IHNwZWNpZmljIHByb3BlcnRpZXMgKGlkZW50aWZp ZWQgYnkgIyBzZXBhcmF0b3IpCisgKgorICogQW4gZXhhbXBsZSBvZiBhIGRhdGFzZXQgc3Bl Y2lmaWMgcHJvcGVydHkgd291bGQgYmUgJ3Bvb2wvZnMjcXVvdGEnCisgKgorICogRGF0YXNl dCBzcGVjaWZpYyBleHRlcm5hbCBwcm9wZXJ0aWVzIHRha2UgcHJlY2lkZW5jZSBvdmVyIG1h dGNoaW5nIGdsb2JhbAorICogcHJvcGVydGllcy4KKyAqLworc3RhdGljIGludAorcHJvcHNf b3ZlcnJpZGUoY2hhciAqZHNuYW1lLCBudmxpc3RfdCAqcHJvcHMsIG52bGlzdF90ICpleHBy b3BzLAorICAgIG52bGlzdF90ICoqbnByb3BzcCwgcmVjdmZsYWdzX3QgKmZsYWdzLCBsaWJ6 ZnNfaGFuZGxlX3QgKmhkbCwKKyAgICB6ZnNfdHlwZV90IHR5cGUsIHVpbnQ2NF90IHpvbmVk LCB6ZnNfaGFuZGxlX3QgKnpocCwKKyAgICBjb25zdCBjaGFyICplcnJidWYpCit7CisJbnZs aXN0X3QgKmRvcHJvcHMsICpnb3Byb3BzLCAqZHhwcm9wcywgKmd4cHJvcHMsICpucHJvcHMs ICp2cHJvcHM7CisJbnZwYWlyX3QgKnBhaXI7CisJY2hhciAqc3RydmFsOworCWludCByZXQg PSAwOworCWNvbnN0IGNoYXIgc2VwID0gJyMnOworCisJaWYgKG52bGlzdF9lbXB0eShwcm9w cykgfHwgbnZsaXN0X2VtcHR5KGV4cHJvcHMpKQorCQlyZXR1cm4gKDApOyAvKiBObyBwcm9w ZXJ0aWVzICovCisKKwlpZiAobnZsaXN0X2R1cChwcm9wcywgJm5wcm9wcywgMCkgIT0gMCkK KwkJcmV0dXJuICgtMSk7CisKKwlWRVJJRlkobnZsaXN0X2FsbG9jKCZkb3Byb3BzLCBOVl9V TklRVUVfTkFNRSwgMCkgPT0gMCk7CisJVkVSSUZZKG52bGlzdF9hbGxvYygmZ29wcm9wcywg TlZfVU5JUVVFX05BTUUsIDApID09IDApOworCVZFUklGWShudmxpc3RfYWxsb2MoJmR4cHJv cHMsIE5WX1VOSVFVRV9OQU1FLCAwKSA9PSAwKTsKKwlWRVJJRlkobnZsaXN0X2FsbG9jKCZn eHByb3BzLCBOVl9VTklRVUVfTkFNRSwgMCkgPT0gMCk7CisKKwkvKiBidWlsZCBsaXN0cyB0 byBwcm9jZXNzIGluIG9yZGVyICovCisJZm9yIChudnBhaXJfdCAqcGFpciA9IG52bGlzdF9u ZXh0X252cGFpcihleHByb3BzLCBOVUxMKTsgcGFpciAhPSBOVUxMOworCSAgICAgcGFpciA9 IG52bGlzdF9uZXh0X252cGFpcihleHByb3BzLCBwYWlyKSkgeworCQljb25zdCBjaGFyICpw cm9wbmFtZSA9IG52cGFpcl9uYW1lKHBhaXIpOworCQljaGFyICpzZXBwID0gc3RyY2hyKHBy b3BuYW1lLCBzZXApOworCQlpZiAoc2VwcCA9PSBOVUxMKSB7CisJCQlzd2l0Y2gobnZwYWly X3R5cGUocGFpcikpCisJCQl7CisJCQljYXNlIERBVEFfVFlQRV9CT09MRUFOOgorCQkJCVZF UklGWTAobnZsaXN0X2FkZF9udnBhaXIoZ3hwcm9wcywgcGFpcikpOworCQkJCWJyZWFrOwor CQkJY2FzZSBEQVRBX1RZUEVfU1RSSU5HOgorCQkJCVZFUklGWTAobnZsaXN0X2FkZF9udnBh aXIoZ29wcm9wcywgcGFpcikpOworCQkJCWJyZWFrOworCQkJZGVmYXVsdDoKKwkJCQkodm9p ZCkgZnByaW50ZihzdGRlcnIsIGRnZXR0ZXh0KFRFWFRfRE9NQUlOLAorCQkJCSAgICAicHJv cCAnJXMnIG11c3QgYmUgYSBzdHJpbmcgb3IgYm9vbGVhbiIpLAorCQkJCSAgICBwcm9wbmFt ZSk7CisJCQkJLyogc2hvdWxkIG5ldmVyIGhhcHBlbiwgc28gYXNzZXJ0ICovCisJCQkJYXNz ZXJ0KEJfRkFMU0UpOworCQkJfQorCQl9IGVsc2UgaWYgKHN0cmNtcChkc25hbWUsIHNlcHAr MSkgPT0gMCkgeworCQkJLyogZGF0YXNldCBzcGVjaWZpYyBwcm9wZXJ0eSAqLworCQkJKnNl cHAgPSAnXDAnOworCQkJc3dpdGNoKG52cGFpcl90eXBlKHBhaXIpKQorCQkJeworCQkJY2Fz ZSBEQVRBX1RZUEVfQk9PTEVBTjoKKwkJCQlWRVJJRlkwKG52bGlzdF9hZGRfYm9vbGVhbihk eHByb3BzLCBwcm9wbmFtZSkpOworCQkJCWJyZWFrOworCQkJY2FzZSBEQVRBX1RZUEVfU1RS SU5HOgorCQkJCW52cGFpcl92YWx1ZV9zdHJpbmcocGFpciwgJnN0cnZhbCk7CisJCQkJVkVS SUZZMChudmxpc3RfYWRkX3N0cmluZyhkb3Byb3BzLCBwcm9wbmFtZSwKKwkJCQkgICAgc3Ry dmFsKSk7CisJCQkJYnJlYWs7CisJCQlkZWZhdWx0OgorCQkJCSh2b2lkKSBmcHJpbnRmKHN0 ZGVyciwgZGdldHRleHQoVEVYVF9ET01BSU4sCisJCQkJICAgICJwcm9wICclcycgbXVzdCBi ZSBhIHN0cmluZyBvciBib29sZWFuIiksCisJCQkJICAgIHByb3BuYW1lKTsKKwkJCQkvKiBz aG91bGQgbmV2ZXIgaGFwcGVuLCBzbyBhc3NlcnQgKi8KKwkJCQlhc3NlcnQoQl9GQUxTRSk7 CisJCQl9CisJCQkqc2VwcCA9IHNlcDsKKwkJfQorCX0KKworCS8qIGNvbnZlcnQgb3ZlcnJp ZGUgcHJvcGVydGllcyBlLmcuIHN0cmluZ3MgdG8gbmF0aXZlICovCisJaWYgKCh2cHJvcHMg PSB6ZnNfdmFsaWRfcHJvcGxpc3QoaGRsLCB0eXBlLCBnb3Byb3BzLCB6b25lZCwgemhwLAor CSAgICBlcnJidWYpKSA9PSBOVUxMKQorCQlnb3RvIGVycm9yOworCisJbnZsaXN0X2ZyZWUo Z29wcm9wcyk7CisJZ29wcm9wcyA9IHZwcm9wczsKKworCWlmICgodnByb3BzID0gemZzX3Zh bGlkX3Byb3BsaXN0KGhkbCwgdHlwZSwgZG9wcm9wcywgem9uZWQsIHpocCwKKwkgICAgZXJy YnVmKSkgPT0gTlVMTCkKKwkJZ290byBlcnJvcjsKKwkKKwludmxpc3RfZnJlZShkb3Byb3Bz KTsKKwlkb3Byb3BzID0gdnByb3BzOworCisJLyogZ2xvYmFsIC0gb3ZlcnJpZGUgLyBzZXQg cHJvcGVydGllcyAqLworCWZvciAobnZwYWlyX3QgKnBhaXIgPSBudmxpc3RfbmV4dF9udnBh aXIoZ29wcm9wcywgTlVMTCk7IHBhaXIgIT0gTlVMTDsKKwkgICAgIHBhaXIgPSBudmxpc3Rf bmV4dF9udnBhaXIoZ29wcm9wcywgcGFpcikpIHsKKwkJY29uc3QgY2hhciAqcG5hbWUgPSBu dnBhaXJfbmFtZShwYWlyKTsKKwkJaWYgKCFudmxpc3RfZXhpc3RzKGd4cHJvcHMsIHBuYW1l KSAmJgorCQkgICAgIW52bGlzdF9leGlzdHMoZHhwcm9wcywgcG5hbWUpKSB7CisJCQlpZiAo ZmxhZ3MtPnZlcmJvc2UpCisJCQkJKHZvaWQpIHByaW50ZigiJXMgJXMgcHJvcGVydHkgZnJv bSAlc1xuIiwKKwkJCQkgICAgbnZsaXN0X2V4aXN0cyhucHJvcHMsIHBuYW1lKSA/CisJCQkJ ICAgICJvdmVycmlkaW5nIiA6ICJzZXR0aW5nIiwgcG5hbWUsIGRzbmFtZSk7CisJCQlWRVJJ RlkwKG52bGlzdF9hZGRfbnZwYWlyKG5wcm9wcywgcGFpcikpOworCQl9CisJfQorCisJLyog Z2xvYmFsIC0gZXhjbHVkZSBwcm9wZXJ0aWVzICovCisJZm9yIChudnBhaXJfdCAqcGFpciA9 IG52bGlzdF9uZXh0X252cGFpcihneHByb3BzLCBOVUxMKTsgcGFpciAhPSBOVUxMOworCSAg ICAgcGFpciA9IG52bGlzdF9uZXh0X252cGFpcihneHByb3BzLCBwYWlyKSkgeworCQljb25z dCBjaGFyICpwbmFtZSA9IG52cGFpcl9uYW1lKHBhaXIpOworCQlpZiAoIW52bGlzdF9leGlz dHMoZG9wcm9wcywgcG5hbWUpKSB7CisJCQlpZiAoZmxhZ3MtPnZlcmJvc2UgJiYgbnZsaXN0 X2V4aXN0cyhucHJvcHMsIHBuYW1lKSkKKwkJCQkodm9pZCkgcHJpbnRmKCJleGNsdWRpbmcg JXMgcHJvcGVydHkgZnJvbSAlc1xuIiwKKwkJCQkgICAgcG5hbWUsIGRzbmFtZSk7CisKKwkJ CSh2b2lkKSBudmxpc3RfcmVtb3ZlX2FsbChucHJvcHMsIHBuYW1lKTsKKwkJfQorCX0KKwor CS8qIGRhdGFzZXQgLSBvdmVycmlkZSAvIHNldCBwcm9wZXJ0aWVzICovCisJZm9yIChudnBh aXJfdCAqcGFpciA9IG52bGlzdF9uZXh0X252cGFpcihkb3Byb3BzLCBOVUxMKTsgcGFpciAh PSBOVUxMOworCSAgICAgcGFpciA9IG52bGlzdF9uZXh0X252cGFpcihkb3Byb3BzLCBwYWly KSkgeworCQljb25zdCBjaGFyICpwbmFtZSA9IG52cGFpcl9uYW1lKHBhaXIpOworCQlpZiAo IW52bGlzdF9leGlzdHMoZHhwcm9wcywgcG5hbWUpKSB7CisJCQlpZiAoZmxhZ3MtPnZlcmJv c2UpCisJCQkJKHZvaWQpIHByaW50ZigiJXMgJXMgcHJvcGVydHkgZnJvbSAlc1xuIiwKKwkJ CQkgICAgbnZsaXN0X2V4aXN0cyhucHJvcHMsIHBuYW1lKSA/CisJCQkJICAgICJvdmVycmlk aW5nIiA6ICJzZXR0aW5nIiwgcG5hbWUsIGRzbmFtZSk7CisJCQlWRVJJRlkwKG52bGlzdF9h ZGRfbnZwYWlyKG5wcm9wcywgcGFpcikpOworCQl9CisJfQorCisJLyogZGF0YXNldCAtIGV4 Y2x1ZGUgcHJvcGVydGllcyAqLworCWZvciAobnZwYWlyX3QgKnBhaXIgPSBudmxpc3RfbmV4 dF9udnBhaXIoZHhwcm9wcywgTlVMTCk7IHBhaXIgIT0gTlVMTDsKKwkgICAgIHBhaXIgPSBu dmxpc3RfbmV4dF9udnBhaXIoZHhwcm9wcywgcGFpcikpIHsKKwkJY29uc3QgY2hhciAqcG5h bWUgPSBudnBhaXJfbmFtZShwYWlyKTsKKwkJaWYgKG52bGlzdF9leGlzdHMobnByb3BzLCBw bmFtZSkpIHsKKwkJCWlmIChmbGFncy0+dmVyYm9zZSkKKwkJCQkodm9pZCkgcHJpbnRmKCJl eGNsdWRpbmcgJXMgcHJvcGVydHkgZnJvbSAlc1xuIiwKKwkJCQkgICAgcG5hbWUsIGRzbmFt ZSk7CisKKwkJCSh2b2lkKSBudmxpc3RfcmVtb3ZlX2FsbChucHJvcHMsIHBuYW1lKTsKKwkJ fQorCX0KKworCSpucHJvcHNwID0gbnByb3BzOworCitlcnJvcjoKKwlpZiAoMCAhPSByZXQp CisJCW52bGlzdF9mcmVlKG5wcm9wcyk7CisJbnZsaXN0X2ZyZWUoZ29wcm9wcyk7CisJbnZs aXN0X2ZyZWUoZ3hwcm9wcyk7CisJbnZsaXN0X2ZyZWUoZG9wcm9wcyk7CisJbnZsaXN0X2Zy ZWUoZHhwcm9wcyk7CisJcmV0dXJuIChyZXQpOworfQorCisvKgogICogUmVzdG9yZXMgYSBi YWNrdXAgb2YgdG9zbmFwIGZyb20gdGhlIGZpbGUgZGVzY3JpcHRvciBzcGVjaWZpZWQgYnkg aW5mZC4KICAqLwogc3RhdGljIGludAogemZzX3JlY2VpdmVfb25lKGxpYnpmc19oYW5kbGVf dCAqaGRsLCBpbnQgaW5mZCwgY29uc3QgY2hhciAqdG9zbmFwLAotICAgIHJlY3ZmbGFnc190 ICpmbGFncywgZG11X3JlcGxheV9yZWNvcmRfdCAqZHJyLAotICAgIGRtdV9yZXBsYXlfcmVj b3JkX3QgKmRycl9ub3N3YXAsIGNvbnN0IGNoYXIgKnNlbmRmcywKLSAgICBudmxpc3RfdCAq c3RyZWFtX252LCBhdmxfdHJlZV90ICpzdHJlYW1fYXZsLCBjaGFyICoqdG9wX3pmcywgaW50 IGNsZWFudXBfZmQsCi0gICAgdWludDY0X3QgKmFjdGlvbl9oYW5kbGVwKQorICAgIHJlY3Zm bGFnc190ICpmbGFncywgbnZsaXN0X3QgKmV4cHJvcHMsIG52bGlzdF90ICpsaW1pdGRzLAor ICAgIGRtdV9yZXBsYXlfcmVjb3JkX3QgKmRyciwgZG11X3JlcGxheV9yZWNvcmRfdCAqZHJy X25vc3dhcCwKKyAgICBjb25zdCBjaGFyICpzZW5kZnMsIG52bGlzdF90ICpzdHJlYW1fbnYs IGF2bF90cmVlX3QgKnN0cmVhbV9hdmwsCisgICAgY2hhciAqKnRvcF96ZnMsIGludCBjbGVh bnVwX2ZkLCB1aW50NjRfdCAqYWN0aW9uX2hhbmRsZXApCiB7CiAJemZzX2NtZF90IHpjID0g eyAwIH07CiAJdGltZV90IGJlZ2luX3RpbWU7CiAJaW50IGlvY3RsX2VyciwgaW9jdGxfZXJy bm8sIGVycjsKIAljaGFyICpjcDsKIAlzdHJ1Y3QgZHJyX2JlZ2luICpkcnJiID0gJmRyci0+ ZHJyX3UuZHJyX2JlZ2luOworCWNoYXIgZHNuYW1lW1pGU19NQVhOQU1FTEVOXTsKIAljaGFy IGVycmJ1ZlsxMDI0XTsKIAljaGFyIHByb3BfZXJyYnVmWzEwMjRdOwogCWNvbnN0IGNoYXIg KmNob3BwcmVmaXg7CiAJYm9vbGVhbl90IG5ld2ZzID0gQl9GQUxTRTsKLQlib29sZWFuX3Qg c3RyZWFtX3dhbnRzbmV3ZnM7CisJYm9vbGVhbl90IHN0cmVhbV93YW50c25ld2ZzLCBza2lw OwogCXVpbnQ2NF90IHBhcmVudF9zbmFwZ3VpZCA9IDA7CiAJcHJvcF9jaGFuZ2VsaXN0X3Qg KmNscCA9IE5VTEw7CiAJbnZsaXN0X3QgKnNuYXBwcm9wc19udmxpc3QgPSBOVUxMOworCW52 bGlzdF90ICpwcm9wcyA9IE5VTEw7CisJbnZsaXN0X3QgKm5wcm9wcyA9IE5VTEw7CiAJenBy b3BfZXJyZmxhZ3NfdCBwcm9wX2VycmZsYWdzOwotCWJvb2xlYW5fdCByZWN1cnNpdmU7CisJ Ym9vbGVhbl90IHJlY3Vyc2l2ZSwgaGFzX2V4cHJvcHM7CiAKIAliZWdpbl90aW1lID0gdGlt ZShOVUxMKTsKIApAQCAtMjY2MiwxMCArMjgzOCw5IEBAIHpmc19yZWNlaXZlX29uZShsaWJ6 ZnNfaGFuZGxlX3QgKmhkbCwgaW4KIAogCWlmIChzdHJlYW1fYXZsICE9IE5VTEwpIHsKIAkJ Y2hhciAqc25hcG5hbWU7CisJCW52bGlzdF90ICpzbmFwcHJvcHM7CiAJCW52bGlzdF90ICpm cyA9IGZzYXZsX2ZpbmQoc3RyZWFtX2F2bCwgZHJyYi0+ZHJyX3RvZ3VpZCwKIAkJICAgICZz bmFwbmFtZSk7Ci0JCW52bGlzdF90ICpwcm9wczsKLQkJaW50IHJldDsKIAogCQkodm9pZCkg bnZsaXN0X2xvb2t1cF91aW50NjQoZnMsICJwYXJlbnRmcm9tc25hcCIsCiAJCSAgICAmcGFy ZW50X3NuYXBndWlkKTsKQEAgLTI2NzcsMTcgKzI4NTIsMTYgQEAgemZzX3JlY2VpdmVfb25l KGxpYnpmc19oYW5kbGVfdCAqaGRsLCBpbgogCQkJVkVSSUZZKDAgPT0gbnZsaXN0X2FkZF91 aW50NjQocHJvcHMsCiAJCQkgICAgemZzX3Byb3BfdG9fbmFtZShaRlNfUFJPUF9DQU5NT1VO VCksIDApKTsKIAkJfQotCQlyZXQgPSB6Y21kX3dyaXRlX3NyY19udmxpc3QoaGRsLCAmemMs IHByb3BzKTsKLQkJaWYgKGVycikKKworCQlpZiAoZXJyKSB7CiAJCQludmxpc3RfZnJlZShw cm9wcyk7CisJCQlwcm9wcyA9IE5VTEw7CisJCX0KIAotCQlpZiAoMCA9PSBudmxpc3RfbG9v a3VwX252bGlzdChmcywgInNuYXBwcm9wcyIsICZwcm9wcykpIHsKLQkJCVZFUklGWSgwID09 IG52bGlzdF9sb29rdXBfbnZsaXN0KHByb3BzLAorCQlpZiAoMCA9PSBudmxpc3RfbG9va3Vw X252bGlzdChmcywgInNuYXBwcm9wcyIsICZzbmFwcHJvcHMpKSB7CisJCQlWRVJJRlkoMCA9 PSBudmxpc3RfbG9va3VwX252bGlzdChzbmFwcHJvcHMsCiAJCQkgICAgc25hcG5hbWUsICZz bmFwcHJvcHNfbnZsaXN0KSk7CiAJCX0KLQotCQlpZiAocmV0ICE9IDApCi0JCQlyZXR1cm4g KC0xKTsKIAl9CiAKIAljcCA9IE5VTEw7CkBAIC0yODU3LDYgKzMwMzEsMTEgQEAgemZzX3Jl Y2VpdmVfb25lKGxpYnpmc19oYW5kbGVfdCAqaGRsLCBpbgogCSh2b2lkKSBzdHJjcHkoemMu emNfbmFtZSwgemMuemNfdmFsdWUpOwogCSpzdHJjaHIoemMuemNfbmFtZSwgJ0AnKSA9ICdc MCc7CiAKKwkodm9pZCkgc3RyY3B5KGRzbmFtZSwgZHJyYi0+ZHJyX3RvbmFtZSk7CisJKnN0 cmNocihkc25hbWUsICdAJykgPSAnXDAnOworCisJaGFzX2V4cHJvcHMgPSAhbnZsaXN0X2Vt cHR5KGV4cHJvcHMpOworCiAJaWYgKHpmc19kYXRhc2V0X2V4aXN0cyhoZGwsIHpjLnpjX25h bWUsIFpGU19UWVBFX0RBVEFTRVQpKSB7CiAJCXpmc19oYW5kbGVfdCAqemhwOwogCkBAIC0y OTIwLDYgKzMwOTksMTYgQEAgemZzX3JlY2VpdmVfb25lKGxpYnpmc19oYW5kbGVfdCAqaGRs LCBpbgogCQkJCXJldHVybiAoLTEpOwogCQkJfQogCQl9CisKKwkJLyogY29udmVydCBvdmVy cmlkZSBwcm9wZXJ0aWVzIGUuZy4gc3RyaW5ncyB0byBuYXRpdmUgKi8KKwkJaWYgKGhhc19l eHByb3BzICYmIHByb3BzX292ZXJyaWRlKGRzbmFtZSwgcHJvcHMsIGV4cHJvcHMsCisJCSAg ICAmbnByb3BzLCBmbGFncywgaGRsLCB6aHAtPnpmc190eXBlLAorCQkgICAgemZzX3Byb3Bf Z2V0X2ludCh6aHAsIFpGU19QUk9QX1pPTkVEKSwgemhwLCBlcnJidWYpICE9IDApIHsKKwkJ CXpmc19jbG9zZSh6aHApOworCQkJemNtZF9mcmVlX252bGlzdHMoJnpjKTsKKwkJCXJldHVy biAoLTEpOworCQl9CisKIAkJemZzX2Nsb3NlKHpocCk7CiAJfSBlbHNlIHsKIAkJLyoKQEAg LTI5NTAsMjAgKzMxMzksMzkgQEAgemZzX3JlY2VpdmVfb25lKGxpYnpmc19oYW5kbGVfdCAq aGRsLCBpbgogCQl9CiAKIAkJbmV3ZnMgPSBCX1RSVUU7CisJCWlmIChoYXNfZXhwcm9wcykg eworCQkJLyogQ3JlYXRlIGFuIG92ZXJyaWRlIHNldCBvZiBwcm9wZXJ0aWVzIGlmIG5lZWRl ZCAqLworCQkJdWludDY0X3Qgem9uZWQgPSAwOworCQkJaWYgKGZsYWdzLT5pc3ByZWZpeCAm JiAhZmxhZ3MtPmlzdGFpbCAmJiAhZmxhZ3MtPmRyeXJ1bikgeworCQkJCS8qIENoZWNrIGlm IHdlJ3JlIHpvbmVkIG9yIG5vdCAqLworCQkJCWlmIChjaGVja19wYXJlbnRzKGhkbCwgemMu emNfdmFsdWUsICZ6b25lZCwgQl9GQUxTRSwgTlVMTCkgIT0gMCkgeworCQkJCQl6Y21kX2Zy ZWVfbnZsaXN0cygmemMpOworCQkJCQlyZXR1cm4gKC0xKTsKKwkJCQl9CisJCQl9CisKKwkJ CWlmIChwcm9wc19vdmVycmlkZShkc25hbWUsIHByb3BzLCBleHByb3BzLCAmbnByb3BzLCBm bGFncywKKwkJCSAgICBoZGwsIFpGU19UWVBFX0RBVEFTRVQsIHpvbmVkLCBOVUxMLCBlcnJi dWYpICE9IDApIHsKKwkJCQl6Y21kX2ZyZWVfbnZsaXN0cygmemMpOworCQkJCXJldHVybiAo LTEpOworCQkJfQorCQl9CiAJfQogCiAJemMuemNfYmVnaW5fcmVjb3JkID0gZHJyX25vc3dh cC0+ZHJyX3UuZHJyX2JlZ2luOwogCXpjLnpjX2Nvb2tpZSA9IGluZmQ7CiAJemMuemNfZ3Vp ZCA9IGZsYWdzLT5mb3JjZTsKKwlza2lwID0gIW52bGlzdF9lbXB0eShsaW1pdGRzKSAmJiAh bnZsaXN0X2V4aXN0cyhsaW1pdGRzLCBkc25hbWUpOwogCWlmIChmbGFncy0+dmVyYm9zZSkg ewogCQkodm9pZCkgcHJpbnRmKCIlcyAlcyBzdHJlYW0gb2YgJXMgaW50byAlc1xuIiwKLQkJ ICAgIGZsYWdzLT5kcnlydW4gPyAid291bGQgcmVjZWl2ZSIgOiAicmVjZWl2aW5nIiwKKwkJ ICAgIHNraXAgPyAoZmxhZ3MtPmRyeXJ1biA/ICJ3b3VsZCBza2lwIiA6ICJza2lwcGluZyIp IDoKKwkJICAgIChmbGFncy0+ZHJ5cnVuID8gIndvdWxkIHJlY2VpdmUiIDogInJlY2Vpdmlu ZyIpLAogCQkgICAgZHJyYi0+ZHJyX2Zyb21ndWlkID8gImluY3JlbWVudGFsIiA6ICJmdWxs IiwKIAkJICAgIGRycmItPmRycl90b25hbWUsIHpjLnpjX3ZhbHVlKTsKIAkJKHZvaWQpIGZm bHVzaChzdGRvdXQpOwogCX0KIAotCWlmIChmbGFncy0+ZHJ5cnVuKSB7CisJaWYgKGZsYWdz LT5kcnlydW4gfHwgc2tpcCkgewogCQl6Y21kX2ZyZWVfbnZsaXN0cygmemMpOwogCQlyZXR1 cm4gKHJlY3Zfc2tpcChoZGwsIGluZmQsIGZsYWdzLT5ieXRlc3dhcCkpOwogCX0KQEAgLTI5 NzMsNiArMzE4MSwxNSBAQCB6ZnNfcmVjZWl2ZV9vbmUobGliemZzX2hhbmRsZV90ICpoZGws IGluCiAJemMuemNfY2xlYW51cF9mZCA9IGNsZWFudXBfZmQ7CiAJemMuemNfYWN0aW9uX2hh bmRsZSA9ICphY3Rpb25faGFuZGxlcDsKIAorCWlmIChucHJvcHMpIHsKKwkJaWYgKHpjbWRf d3JpdGVfc3JjX252bGlzdChoZGwsICZ6YywgbnByb3BzKSAhPSAwKSB7CisJCQludmxpc3Rf ZnJlZShucHJvcHMpOworCQkJcmV0dXJuICgtMSk7CisJCX0KKwkJbnZsaXN0X2ZyZWUobnBy b3BzKTsKKwl9IGVsc2UgaWYgKHByb3BzICYmIHpjbWRfd3JpdGVfc3JjX252bGlzdChoZGws ICZ6YywgcHJvcHMpICE9IDApCisJCXJldHVybiAoLTEpOworCiAJZXJyID0gaW9jdGxfZXJy ID0gemZzX2lvY3RsKGhkbCwgWkZTX0lPQ19SRUNWLCAmemMpOwogCWlvY3RsX2Vycm5vID0g ZXJybm87CiAJcHJvcF9lcnJmbGFncyA9ICh6cHJvcF9lcnJmbGFnc190KXpjLnpjX29iajsK QEAgLTMxODAsOCArMzM5Nyw5IEBAIHpmc19yZWNlaXZlX29uZShsaWJ6ZnNfaGFuZGxlX3Qg KmhkbCwgaW4KIAogc3RhdGljIGludAogemZzX3JlY2VpdmVfaW1wbChsaWJ6ZnNfaGFuZGxl X3QgKmhkbCwgY29uc3QgY2hhciAqdG9zbmFwLCByZWN2ZmxhZ3NfdCAqZmxhZ3MsCi0gICAg aW50IGluZmQsIGNvbnN0IGNoYXIgKnNlbmRmcywgbnZsaXN0X3QgKnN0cmVhbV9udiwgYXZs X3RyZWVfdCAqc3RyZWFtX2F2bCwKLSAgICBjaGFyICoqdG9wX3pmcywgaW50IGNsZWFudXBf ZmQsIHVpbnQ2NF90ICphY3Rpb25faGFuZGxlcCkKKyAgICBpbnQgaW5mZCwgbnZsaXN0X3Qg KmV4cHJvcHMsIG52bGlzdF90ICpsaW1pdGRzLCBjb25zdCBjaGFyICpzZW5kZnMsCisgICAg bnZsaXN0X3QgKnN0cmVhbV9udiwgYXZsX3RyZWVfdCAqc3RyZWFtX2F2bCwgY2hhciAqKnRv cF96ZnMsIGludCBjbGVhbnVwX2ZkLAorICAgIHVpbnQ2NF90ICphY3Rpb25faGFuZGxlcCkK IHsKIAlpbnQgZXJyOwogCWRtdV9yZXBsYXlfcmVjb3JkX3QgZHJyLCBkcnJfbm9zd2FwOwpA QCAtMzI3MywxMyArMzQ5MSwxNCBAQCB6ZnNfcmVjZWl2ZV9pbXBsKGxpYnpmc19oYW5kbGVf dCAqaGRsLCBjCiAJCQlzZW5kZnMgPSBub25wYWNrYWdlX3NlbmRmczsKIAkJfQogCQlyZXR1 cm4gKHpmc19yZWNlaXZlX29uZShoZGwsIGluZmQsIHRvc25hcCwgZmxhZ3MsCi0JCSAgICAm ZHJyLCAmZHJyX25vc3dhcCwgc2VuZGZzLCBzdHJlYW1fbnYsIHN0cmVhbV9hdmwsCi0JCSAg ICB0b3BfemZzLCBjbGVhbnVwX2ZkLCBhY3Rpb25faGFuZGxlcCkpOworCQkgICAgZXhwcm9w cywgbGltaXRkcywgJmRyciwgJmRycl9ub3N3YXAsIHNlbmRmcywgc3RyZWFtX252LAorCQkg ICAgc3RyZWFtX2F2bCwgdG9wX3pmcywgY2xlYW51cF9mZCwgYWN0aW9uX2hhbmRsZXApKTsK IAl9IGVsc2UgewogCQlhc3NlcnQoRE1VX0dFVF9TVFJFQU1fSERSVFlQRShkcnJiLT5kcnJf dmVyc2lvbmluZm8pID09CiAJCSAgICBETVVfQ09NUE9VTkRTVFJFQU0pOwogCQlyZXR1cm4g KHpmc19yZWNlaXZlX3BhY2thZ2UoaGRsLCBpbmZkLCB0b3NuYXAsIGZsYWdzLAotCQkgICAg JmRyciwgJnpja3N1bSwgdG9wX3pmcywgY2xlYW51cF9mZCwgYWN0aW9uX2hhbmRsZXApKTsK KwkJICAgIGV4cHJvcHMsIGxpbWl0ZHMsICZkcnIsICZ6Y2tzdW0sIHRvcF96ZnMsIGNsZWFu dXBfZmQsCisJCSAgICBhY3Rpb25faGFuZGxlcCkpOwogCX0KIH0KIApAQCAtMzI5MSw3ICsz NTEwLDcgQEAgemZzX3JlY2VpdmVfaW1wbChsaWJ6ZnNfaGFuZGxlX3QgKmhkbCwgYwogICov CiBpbnQKIHpmc19yZWNlaXZlKGxpYnpmc19oYW5kbGVfdCAqaGRsLCBjb25zdCBjaGFyICp0 b3NuYXAsIHJlY3ZmbGFnc190ICpmbGFncywKLSAgICBpbnQgaW5mZCwgYXZsX3RyZWVfdCAq c3RyZWFtX2F2bCkKKyAgICBpbnQgaW5mZCwgbnZsaXN0X3QgKmV4cHJvcHMsIG52bGlzdF90 ICpsaW1pdGRzLCBhdmxfdHJlZV90ICpzdHJlYW1fYXZsKQogewogCWNoYXIgKnRvcF96ZnMg PSBOVUxMOwogCWludCBlcnI7CkBAIC0zMzAxLDggKzM1MjAsOCBAQCB6ZnNfcmVjZWl2ZShs aWJ6ZnNfaGFuZGxlX3QgKmhkbCwgY29uc3QgCiAJY2xlYW51cF9mZCA9IG9wZW4oWkZTX0RF ViwgT19SRFdSfE9fRVhDTCk7CiAJVkVSSUZZKGNsZWFudXBfZmQgPj0gMCk7CiAKLQllcnIg PSB6ZnNfcmVjZWl2ZV9pbXBsKGhkbCwgdG9zbmFwLCBmbGFncywgaW5mZCwgTlVMTCwgTlVM TCwKLQkgICAgc3RyZWFtX2F2bCwgJnRvcF96ZnMsIGNsZWFudXBfZmQsICZhY3Rpb25faGFu ZGxlKTsKKwllcnIgPSB6ZnNfcmVjZWl2ZV9pbXBsKGhkbCwgdG9zbmFwLCBmbGFncywgaW5m ZCwgZXhwcm9wcywgbGltaXRkcywgTlVMTCwKKwkgICAgTlVMTCwgc3RyZWFtX2F2bCwgJnRv cF96ZnMsIGNsZWFudXBfZmQsICZhY3Rpb25faGFuZGxlKTsKIAogCVZFUklGWSgwID09IGNs b3NlKGNsZWFudXBfZmQpKTsKIAotLS0gY2RkbC9jb250cmliL29wZW5zb2xhcmlzL2xpYi9s aWJ6ZnMvY29tbW9uL2xpYnpmcy5oLm9yaWcJMjAxNC0xMC0xMCAwOTozMTozNi4wMDAwMDAw MDAgKzAwMDAKKysrIGNkZGwvY29udHJpYi9vcGVuc29sYXJpcy9saWIvbGliemZzL2NvbW1v bi9saWJ6ZnMuaAkyMDE0LTExLTEzIDE0OjMxOjEwLjQ1NjY2OTQwOCArMDAwMApAQCAtNjY2 LDcgKzY2Niw3IEBAIHR5cGVkZWYgc3RydWN0IHJlY3ZmbGFncyB7CiB9IHJlY3ZmbGFnc190 OwogCiBleHRlcm4gaW50IHpmc19yZWNlaXZlKGxpYnpmc19oYW5kbGVfdCAqLCBjb25zdCBj aGFyICosIHJlY3ZmbGFnc190ICosCi0gICAgaW50LCBhdmxfdHJlZV90ICopOworICAgIGlu dCwgbnZsaXN0X3QgKiwgbnZsaXN0X3QgKiwgYXZsX3RyZWVfdCAqKTsKIAogdHlwZWRlZiBl bnVtIGRpZmZfZmxhZ3MgewogCVpGU19ESUZGX1BBUlNFQUJMRSA9IDB4MSwKLS0tIGNkZGwv Y29udHJpYi9vcGVuc29sYXJpcy9jbWQvemZzL3pmcy44Lm9yaWcJMjAxNC0xMC0xMCAwOToz MTozNS4wMDAwMDAwMDAgKzAwMDAKKysrIGNkZGwvY29udHJpYi9vcGVuc29sYXJpcy9jbWQv emZzL3pmcy44CTIwMTQtMTEtMTMgMTQ6MzQ6MDUuMDUwNjU4MDgzICswMDAwCkBAIC0xOTAs MTAgKzE5MCwyMiBAQAogLk5tCiAuQ20gcmVjZWl2ZSBOcyB8IE5zIENtIHJlY3YKIC5PcCBG bCB2bkZ1CisuT3AgRmwgbCBBciBmaWxlc3lzdGVtIE5zIHwgTnMgQXIgdm9sdW1lCisuQXIg Li4uCisuT3AgRmwgbyBBciBwcm9wZXJ0eSBOcyA9IE5zIEFyIHZhbHVlCisuQXIgLi4uCisu T3AgRmwgeCBBciBwcm9wZXJ0eQorLkFyIC4uLgogLkFyIGZpbGVzeXN0ZW0gTnMgfCBOcyBB ciB2b2x1bWUgTnMgfCBOcyBBciBzbmFwc2hvdAogLk5tCiAuQ20gcmVjZWl2ZSBOcyB8IE5z IENtIHJlY3YKIC5PcCBGbCB2bkZ1CisuT3AgRmwgbCBBciBmaWxlc3lzdGVtIE5zIHwgTnMg QXIgdm9sdW1lCisuQXIgLi4uCisuT3AgRmwgbyBBciBwcm9wZXJ0eT12YWx1ZQorLkFyIC4u LgorLk9wIEZsIHggQXIgcHJvcGVydHkKKy5BciAuLi4KIC5PcCBGbCBkIHwgZQogLkFyIGZp bGVzeXN0ZW0KIC5ObQpAQCAtMjY1MCwxMiArMjY2MiwyNCBAQCBmZWF0dXJlLgogLk5tCiAu Q20gcmVjZWl2ZSBOcyB8IE5zIENtIHJlY3YKIC5PcCBGbCB2bkZ1CisuT3AgRmwgbCBBciBm aWxlc3lzdGVtIE5zIHwgTnMgQXIgdm9sdW1lCisuQXIgLi4uCisuT3AgRmwgbyBBciBwcm9w ZXJ0eSBOcyA9IE5zIEFyIHZhbHVlCisuQXIgLi4uCisuT3AgRmwgeCBBciBwcm9wZXJ0eQor LkFyIC4uLgogLkFyIGZpbGVzeXN0ZW0gTnMgfCBOcyBBciB2b2x1bWUgTnMgfCBOcyBBciBz bmFwc2hvdAogLlhjCiAuSXQgWG8KIC5ObQogLkNtIHJlY2VpdmUgTnMgfCBOcyBDbSByZWN2 CiAuT3AgRmwgdm5GdQorLk9wIEZsIGwgQXIgZmlsZXN5c3RlbSBOcyB8IE5zIEFyIHZvbHVt ZQorLkFyIC4uLgorLk9wIEZsIG8gQXIgcHJvcGVydHkgTnMgPSBOcyBBciB2YWx1ZQorLkFy IC4uLgorLk9wIEZsIHggQXIgcHJvcGVydHkKKy5BciAuLi4KIC5PcCBGbCBkIHwgZQogLkFy IGZpbGVzeXN0ZW0KIC5YYwpAQCAtMjc0Nyw2ICsyNzcxLDEwOSBAQCBwZXJmb3JtaW5nIHRo ZSByZWNlaXZlIG9wZXJhdGlvbi4gSWYgcmVjCiBzdHJlYW0gKGZvciBleGFtcGxlLCBvbmUg Z2VuZXJhdGVkIGJ5CiAuUXEgTm0gQ20gc2VuZCBGbCBSIEJybyBGbCBpIHwgRmwgSSBCcmMg KSAsCiBkZXN0cm95IHNuYXBzaG90cyBhbmQgZmlsZSBzeXN0ZW1zIHRoYXQgZG8gbm90IGV4 aXN0IG9uIHRoZSBzZW5kaW5nIHNpZGUuCisuSXQgRmwgbAorTGltaXRzIHRoZSB0aGUgcmVj ZWl2ZSB0byBvbmx5IHRoZQorLkFyIGZpbGVzeXN0ZW0KK29yCisuQXIgdm9sdW1lCitzcGVj aWZpZWQuIEFzIG11bHRpcGxlCisuRmwgbAorb3B0aW9ucyBtYXkgYmUgc3BlY2lmaWVkLCB0 aGlzIGNhbiBiZSB1c2VkIHRvIHJlc3RvcmUgc3BlY2lmaWMgZmlsZXN5c3RlbXMgb3IKK3Zv bHVtZXMgZnJvbSB0aGUgcmVjZWl2ZWQgc3RyZWFtLgorLkl0IEZsIG8gQXIgcHJvcGVydHkg TnMgPSBOcyBBciB2YWx1ZQorU2V0cyB0aGUgc3BlY2lmaWVkIHByb3BlcnR5IGFzIGlmIHRo ZSBjb21tYW5kIHpmcyBzZXQKKy5BciBwcm9wZXJ0eT12YWx1ZQoraXMgaW52b2tlZCBhdCB0 aGUgc2FtZSB0aW1lIHRoZSByZWNlaXZlZCBkYXRhc2V0IGlzIGNyZWF0ZWQgZnJvbSB0aGUK K25vbi1pbmNyZW1lbnRhbCBzZW5kIHN0cmVhbSBvciB1cGRhdGVkIGZyb20gdGhlIGluY3Jl bWVudGFsIHNlbmQgc3RyZWFtLgorLlBwCitBbnkgZWRpdGFibGUgWkZTIHByb3BlcnR5IGNh biBhbHNvIGJlIHNldCBhdCByZWNlaXZlIHRpbWUuIFNldC1vbmNlIHByb3BlcnRpZXMKK2Jv dW5kIHRvIHRoZSByZWNlaXZlZCBkYXRhLCBzdWNoIGFzIG5vcm1hbGl6YXRpb24gYW5kIGNh c2VzZW5zaXRpdml0eSwgY2Fubm90CitiZSBzZXQgYXQgcmVjZWl2ZSB0aW1lIGV2ZW4gd2hl biB0aGUgZGF0YXNldHMgYXJlIG5ld2x5IGNyZWF0ZWQgYnkgemZzIHJlY2VpdmUuCisuUHAK K011bHRpcGxlCisuRmwgbworb3B0aW9ucyBjYW4gYmUgc3BlY2lmaWVkLgorLlBwCitUaGUK Ky5BciBwcm9wZXJ0eQorb3B0aW9uIG1heSB0YWtlIG9uZSBvZiB0d28gZm9ybXMKKy5CbCAt YnVsbGV0IC1jb21wYWN0CisuSXQKKy5BciBwcm9wZXJ0eQorLSBHbG9iYWwgcHJvcGVydHkg YXBwbGllZCB0byBhbGwgc3RyZWFtcy4KKy5JdAorLkFyIHByb3BlcnR5IE5zICMgTnMgT3Ag QXIgdm9sdW1lIE5zIHwgTnMgQXIgZmlsZXN5c3RlbQorLSBMb2NhbCBwcm9wZXJ0eSBhcHBs aWVkIHRvIHRoZSBzcGVjaWZpZWQKKy5BciB2b2x1bWUKK29yIAorLkFyIGZpbGVzeXN0ZW0K K3N0cmVhbXMgb25seS4KKy5FbAorLlBwCitUaGUgbW9zdCBzcGVjaWZpYworLkZsIG8KK29w dGlvbiB0YWtlcyBwcmVjZWRlbmNlIHNvIGluIHRoZSBjYXNlIHdoZXJlIGJvdGggYSBnbG9i YWwKKy5BciBwcm9wZXJ0eQorYW5kIGEKKy5BciBwcm9wZXJ0eSBOcyAjIE5zIE9wIEFyIGZp bGVzeXN0ZW0gTnMgfCBOcyBBciB2b2x1bWUKK2FyZSBzcGVjaWZpZWQgZm9yIHRoZSBzYW1l CisuQXIgcHJvcGVydHkKK3RoZSB2YWx1ZSBvZiBzYWlkCisuQXIgcHJvcGVydHkKK3dpbGwg YmUgdGhlIG9uZSB3aGljaCBtb3N0IGNsb3NlbHkgbWF0Y2hlcyB0aGUgZG9tYWluIG9mIHRo ZQorLkFyIHByb3BlcnR5IC4KKy5QcAorSWYgYm90aAorLkZsIG8KK2FuZCAKKy5GbCB4Cith cmUgc3BlY2lmaWVkIGZvciB0aGUgc2FtZQorLkFyIHByb3BlcnR5Cit0aGUKKy5GbCB4Citv cHRpb24gdGFrZXMgcHJlY2VkZW5jZSB1bmxlc3MgdGhlCisuRmwgbworb3B0aW9uIGlzIGEg YmV0dGVyIGRvbWFpbiBtYXRjaCB0aGFuIHRoZSAKKy5GbCB4CitvcHRpb24uCisuSXQgWG8K Ky5GbCB4IEFyIHByb3BlcnR5CisuWGMKK0Vuc3VyZXMgdGhhdCB0aGUgZWZmZWN0aXZlIHZh bHVlIG9mIHRoZSBzcGVjaWZpZWQgcHJvcGVydHkgYWZ0ZXIgdGhlIHJlY2VpdmUgaXMKK3Vu YWZmZWN0ZWQgYnkgdGhlIHZhbHVlIG9mIHRoYXQgcHJvcGVydHkgaW4gdGhlIHNlbmQgc3Ry ZWFtIChpZiBhbnkpLCBhcyBpZiB0aGUKK3Byb3BlcnR5IGhhZCBiZWVuIGV4Y2x1ZGVkIGZy b20gdGhlIHNlbmQgc3RyZWFtLgorLlBwCitJZiB0aGUgc3BlY2lmaWVkIHByb3BlcnR5IGlz IG5vdCBwcmVzZW50IGluIHRoZSBzZW5kIHN0cmVhbSwgdGhpcyBvcHRpb24gZG9lcworbm90 aGluZy4KKy5QcAorSWYgYSByZWNlaXZlZCBwcm9wZXJ0eSBuZWVkcyB0byBiZSBvdmVycmlk ZGVuLCB0aGUgZWZmZWN0aXZlIHZhbHVlIHdpbGwgYmUKK3NldCBvciBpbmhlcml0ZWQsIGRl cGVuZGluZyBvbiB0aGUgcHJvcGVydHkuCisuUHAKK0luIHRoZSBjYXNlIG9mIGFuIGluY3Jl bWVudGFsIHVwZGF0ZSwKKy5GbCB4CitsZWF2ZXMgYW55IGV4aXN0aW5nIGxvY2FsIHNldHRp bmcgb3IgZXhwbGljaXQgaW5oZXJpdGFuY2UgdW5jaGFuZ2VkIChzaW5jZSB0aGUKK3JlY2Vp dmVkIHByb3BlcnR5IGlzIGFscmVhZHkgb3ZlcnJpZGRlbikuCisuUHAKK1RoZQorLkFyIHBy b3BlcnR5CitvcHRpb24gbWF5IHRha2Ugb25lIG9mIHR3byBmb3JtcworLkJsIC1idWxsZXQg LWNvbXBhY3QKKy5JdAorLkFyIHByb3BlcnR5CistIEdsb2JhbCBwcm9wZXJ0eSBleGNsdWRl ZCBmcm9tIGFsbCBzdHJlYW1zLgorLkl0CisuQXIgcHJvcGVydHkgTnMgIyBOcyBPcCBBciB2 b2x1bWUgTnMgfCBOcyBBciBmaWxlc3lzdGVtCistIExvY2FsIHByb3BlcnR5IGV4Y2x1ZGVk IGZyb20gdGhlIHNwZWNpZmllZAorLkFyIHZvbHVtZQorb3IgCisuQXIgZmlsZXN5c3RlbQor c3RyZWFtcyBvbmx5LgorLkVsCisuUHAKK0FsbAorLkZsIG8KK3Jlc3RyaWN0aW9ucyBhcHBs eSBlcXVhbGx5IHRvCisuRmwgeC4KIC5FbAogLkl0IFhvCiAuTm0KLS0tIGNkZGwvY29udHJp Yi9vcGVuc29sYXJpcy9jbWQvemZzL3pmc19tYWluLmMub3JpZwkyMDE0LTEwLTEwIDA5OjMx OjM1LjAwMDAwMDAwMCArMDAwMAorKysgY2RkbC9jb250cmliL29wZW5zb2xhcmlzL2NtZC96 ZnMvemZzX21haW4uYwkyMDE0LTExLTEzIDE0OjQxOjA3LjE5MTYyOTgwNyArMDAwMApAQCAt MjYyLDkgKzI2MiwxMCBAQCBnZXRfdXNhZ2UoemZzX2hlbHBfdCBpZHgpCiAJY2FzZSBIRUxQ X1BST01PVEU6CiAJCXJldHVybiAoZ2V0dGV4dCgiXHRwcm9tb3RlIDxjbG9uZS1maWxlc3lz dGVtPlxuIikpOwogCWNhc2UgSEVMUF9SRUNFSVZFOgotCQlyZXR1cm4gKGdldHRleHQoIlx0 cmVjZWl2ZXxyZWN2IFstdm5GdV0gPGZpbGVzeXN0ZW18dm9sdW1lfCIKLQkJInNuYXBzaG90 PlxuIgotCQkiXHRyZWNlaXZlfHJlY3YgWy12bkZ1XSBbLWQgfCAtZV0gPGZpbGVzeXN0ZW0+ XG4iKSk7CisJCXJldHVybiAoZ2V0dGV4dCgiXHRyZWNlaXZlfHJlY3YgWy12bkZ1XSBbLW8g PHByb3BlcnR5Pl0gLi4uICIKKwkJICAgICJbLXggPHByb3BlcnR5Pl0gLi4uIDxmaWxlc3lz dGVtfHZvbHVtZXxzbmFwc2hvdD5cbiIKKwkJICAgICJcdHJlY2VpdmV8cmVjdiBbLXZuRnVd IFstZCB8IC1lXSBbLW8gPHByb3BlcnR5Pl0gLi4uICIKKyAgICAgICAgICAgICAgICAgICAg IlsteCA8cHJvcGVydHk+XSAuLi4gPGZpbGVzeXN0ZW0+XG4iKSk7CiAJY2FzZSBIRUxQX1JF TkFNRToKIAkJcmV0dXJuIChnZXR0ZXh0KCJcdHJlbmFtZSBbLWZdIDxmaWxlc3lzdGVtfHZv bHVtZXxzbmFwc2hvdD4gIgogCQkgICAgIjxmaWxlc3lzdGVtfHZvbHVtZXxzbmFwc2hvdD5c biIKQEAgLTQ5NSw2ICs0OTYsMTkgQEAgdXNhZ2UoYm9vbGVhbl90IHJlcXVlc3RlZCkKIH0K IAogc3RhdGljIGludAorYWRkX3VuaXF1ZV9vcHRpb24obnZsaXN0X3QgKm9wdHMsIGNvbnN0 IGNoYXIgKnR5cGUsIGNoYXIgKm5hbWUpCit7CisJaWYgKG52bGlzdF9sb29rdXBfc3RyaW5n KG9wdHMsIG5hbWUsIE5VTEwpID09IDApIHsKKwkJKHZvaWQpIGZwcmludGYoc3RkZXJyLCBn ZXR0ZXh0KCIlcyBvcHRpb24gJyVzJyAiCisJCSAgICAic3BlY2lmaWVkIG11bHRpcGxlIHRp bWVzXG4iKSwgdHlwZSwgbmFtZSk7CisJCXJldHVybiAoLTEpOworCX0KKwlpZiAobnZsaXN0 X2FkZF9ib29sZWFuKG9wdHMsIG5hbWUpKQorCQlub21lbSgpOworCXJldHVybiAoMCk7Cit9 CisKK3N0YXRpYyBpbnQKIHBhcnNlcHJvcChudmxpc3RfdCAqcHJvcHMsIGNoYXIgKnByb3Bu YW1lKQogewogCWNoYXIgKnByb3B2YWwsICpzdHJ2YWw7CkBAIC0zODgyLDE4ICszODk2LDI0 IEBAIHpmc19kb19zZW5kKGludCBhcmdjLCBjaGFyICoqYXJndikKIH0KIAogLyoKLSAqIHpm cyByZWNlaXZlIFstdm5GdV0gWy1kIHwgLWVdIDxmc0BzbmFwPgorICogemZzIHJlY2VpdmUg Wy12bkZ1XSBbLWQgfCAtZV0gWy1sIDx2b2x1bWV8ZmlsZXN5c3RlbT5dIC4uLgorICogWy1v IHByb3BlcnR5PXZhbHVlXSAuLi4gWy14IHByb3BlcnR5XSAuLi4gPHZvbHVtZXxmaWxlc3l0 ZW18c25hcHNob3Q+CiAgKgogICogUmVzdG9yZSBhIGJhY2t1cCBzdHJlYW0gZnJvbSBzdGRp bi4KICAqLwogc3RhdGljIGludAogemZzX2RvX3JlY2VpdmUoaW50IGFyZ2MsIGNoYXIgKiph cmd2KQogeworCW52bGlzdF90ICpleHByb3BzLCAqbGltaXRkczsKIAlpbnQgYywgZXJyOwog CXJlY3ZmbGFnc190IGZsYWdzID0geyAwIH07CisJaWYgKG52bGlzdF9hbGxvYygmZXhwcm9w cywgTlZfVU5JUVVFX05BTUUsIDApICE9IDApCisJCW5vbWVtKCk7CisJaWYgKG52bGlzdF9h bGxvYygmbGltaXRkcywgTlZfVU5JUVVFX05BTUUsIDApICE9IDApCisJCW5vbWVtKCk7CiAK IAkvKiBjaGVjayBvcHRpb25zICovCi0Jd2hpbGUgKChjID0gZ2V0b3B0KGFyZ2MsIGFyZ3Ys ICI6ZGVudXZGIikpICE9IC0xKSB7CisJd2hpbGUgKChjID0gZ2V0b3B0KGFyZ2MsIGFyZ3Ys ICI6ZGVsOm5vOnV2eDpGIikpICE9IC0xKSB7CiAJCXN3aXRjaCAoYykgewogCQljYXNlICdk JzoKIAkJCWZsYWdzLmlzcHJlZml4ID0gQl9UUlVFOwpAQCAtMzkwMiwxNSArMzkyMiwzMiBA QCB6ZnNfZG9fcmVjZWl2ZShpbnQgYXJnYywgY2hhciAqKmFyZ3YpCiAJCQlmbGFncy5pc3By ZWZpeCA9IEJfVFJVRTsKIAkJCWZsYWdzLmlzdGFpbCA9IEJfVFJVRTsKIAkJCWJyZWFrOwor CQljYXNlICdsJzoKKwkJCWlmIChhZGRfdW5pcXVlX29wdGlvbihsaW1pdGRzLCAibGltaXQi LCBvcHRhcmcpKSB7CisJCQkJZXJyID0gMTsKKwkJCQlnb3RvIHJlY3ZlcnJvcjsKKwkJCX0K KwkJCWJyZWFrOwogCQljYXNlICduJzoKIAkJCWZsYWdzLmRyeXJ1biA9IEJfVFJVRTsKIAkJ CWJyZWFrOworCQljYXNlICdvJzoKKwkJCWlmIChwYXJzZXByb3AoZXhwcm9wcywgb3B0YXJn KSkgeworCQkJCWVyciA9IDE7CisJCQkJZ290byByZWN2ZXJyb3I7CisJCQl9CisJCQlicmVh azsKIAkJY2FzZSAndSc6CiAJCQlmbGFncy5ub21vdW50ID0gQl9UUlVFOwogCQkJYnJlYWs7 CiAJCWNhc2UgJ3YnOgogCQkJZmxhZ3MudmVyYm9zZSA9IEJfVFJVRTsKIAkJCWJyZWFrOwor CQljYXNlICd4JzoKKwkJCWlmIChhZGRfdW5pcXVlX29wdGlvbihleHByb3BzLCAiZXhjbHVk ZSIsIG9wdGFyZykpIHsKKwkJCQllcnIgPSAxOworCQkJCWdvdG8gcmVjdmVycm9yOworCQkJ fQogCQljYXNlICdGJzoKIAkJCWZsYWdzLmZvcmNlID0gQl9UUlVFOwogCQkJYnJlYWs7CkBA IC0zOTQ3LDcgKzM5ODQsMTEgQEAgemZzX2RvX3JlY2VpdmUoaW50IGFyZ2MsIGNoYXIgKiph cmd2KQogCQlyZXR1cm4gKDEpOwogCX0KIAotCWVyciA9IHpmc19yZWNlaXZlKGdfemZzLCBh cmd2WzBdLCAmZmxhZ3MsIFNURElOX0ZJTEVOTywgTlVMTCk7CisJZXJyID0gemZzX3JlY2Vp dmUoZ196ZnMsIGFyZ3ZbMF0sICZmbGFncywgU1RESU5fRklMRU5PLCBleHByb3BzLCBsaW1p dGRzLCBOVUxMKTsKKworcmVjdmVycm9yOgorCW52bGlzdF9mcmVlKGV4cHJvcHMpOworCW52 bGlzdF9mcmVlKGxpbWl0ZHMpOwogCiAJcmV0dXJuIChlcnIgIT0gMCk7CiB9Cg== --------------060909070708000505060205-- From owner-freebsd-fs@freebsd.org Fri Feb 26 12:43:17 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 565D6AB4E61 for ; Fri, 26 Feb 2016 12:43:17 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 17A5615F9 for ; Fri, 26 Feb 2016 12:43:16 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 90E1A9DDF8D; Fri, 26 Feb 2016 13:34:40 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: ZFS receive and attributes, proposing some changes. was Re: ZFS full system backup hoses the backup host. From: Borja Marcos In-Reply-To: <56D040E1.3090000@multiplay.co.uk> Date: Fri, 26 Feb 2016 13:34:40 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3D3D7CC4-5E22-4DD0-B935-9EC6BCECB8C3@sarenet.es> References: <0ae0e3ab-6ad1-4ae2-9ee9-a7d993088a01@email.android.com> <56D040E1.3090000@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 12:43:17 -0000 > On 26 Feb 2016, at 13:11, Steven Hartland = wrote: >=20 > I actually have a patch (see attached) which does this but never got = round to finishing the RTI so its got some rough edges. Sounds great, checking it. :) I=E2=80=99m not that familiar with the ZFS code but this looks like the = Perfect Excuse to do it finally! The -o option is busy actually, that=E2=80=99s why I suggested -O. Borja. From owner-freebsd-fs@freebsd.org Fri Feb 26 12:52:25 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3251AB534B for ; Fri, 26 Feb 2016 12:52:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E23CC1DB1 for ; Fri, 26 Feb 2016 12:52:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1QCqO6c097196 for ; Fri, 26 Feb 2016 12:52:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Fri, 26 Feb 2016 12:52:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: johan@300.nl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 12:52:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 johans changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |johan@300.nl --- Comment #15 from johans --- We've just ran into the same bug on one of our machines. # procstat -kk 52827 PID TID COMM TDNAME KSTACK=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 52827 100187 ruby21 - mi_switch+0xe1 thread_suspend_switch+0x170 thread_single+0x4e5 exit1+0xbe sigexit+0x925 postsig+0x286 ast+0x427 doreti_ast+0x1f=20 52827 100389 ruby21 - mi_switch+0xe1 sleepq_wait+0= x3a _sleep+0x287 vnode_create_vobject+0x100 zfs_freebsd_open+0xf5 VOP_OPEN_APV+= 0xa1 vn_open_vnode+0x234 vn_open_cred+0x33e kern_openat+0x26f amd64_syscall+0x357 Xfast_syscall+0xfb=20 (kgdb) print *object $10 =3D { lock =3D { lock_object =3D { lo_name =3D 0xffffffff80ff8196 "vm object",=20 lo_flags =3D 90374144,=20 lo_data =3D 0,=20 lo_witness =3D 0x0 },=20 rw_lock =3D 1 },=20 object_list =3D { tqe_next =3D 0xfffff80135aaa600,=20 tqe_prev =3D 0xfffff80135aaa420 },=20 shadow_head =3D { lh_first =3D 0x0 },=20 shadow_list =3D { le_next =3D 0x0,=20 le_prev =3D 0xfffff8019f85f030 },=20 memq =3D { tqh_first =3D 0x0,=20 tqh_last =3D 0xfffff80135aaa548 },=20 rtree =3D { rt_root =3D 0,=20 rt_flags =3D 0 '\0' },=20 size =3D 0,=20 generation =3D 1,=20 ref_count =3D 0,=20 shadow_count =3D 0,=20 memattr =3D 6 '\006',=20 type =3D 2 '\002',=20 flags =3D 24584,=20 pg_color =3D 1569,=20 paging_in_progress =3D 0,=20 resident_page_count =3D 0,=20 backing_object =3D 0x0,=20 backing_object_offset =3D 0,=20 pager_object_list =3D { tqe_next =3D 0x0,=20 tqe_prev =3D 0x0 },=20 rvq =3D { lh_first =3D 0x0 },=20 cache =3D { rt_root =3D 0,=20 rt_flags =3D 0 '\0' },=20 handle =3D 0xfffff8010bc08760,=20 un_pager =3D { vnp =3D { vnp_size =3D 0,=20 writemappings =3D 0 },=20 devp =3D { devp_pglist =3D { tqh_first =3D 0x0,=20 tqh_last =3D 0x0 },=20 ops =3D 0x0,=20 dev =3D 0x0 },=20 sgp =3D { sgp_pglist =3D { tqh_first =3D 0x0,=20 tqh_last =3D 0x0 } },=20 swp =3D { swp_tmpfs =3D 0x0,=20 swp_bcount =3D 0 } },=20 cred =3D 0x0,=20 charge =3D 0 } (kgdb) p *vp $11 =3D { v_tag =3D 0xffffffff81acefb6 "zfs",=20 v_op =3D 0xffffffff81ae12f0,=20 v_data =3D 0xfffff80031109450,=20 v_mount =3D 0xfffff8000801a330,=20 v_nmntvnodes =3D { tqe_next =3D 0xfffff801c912a1d8,=20 tqe_prev =3D 0xfffff8022bfa0780 },=20 v_un =3D { vu_mount =3D 0x0,=20 vu_socket =3D 0x0,=20 vu_cdev =3D 0x0,=20 vu_fifoinfo =3D 0x0 },=20 v_hashlist =3D { le_next =3D 0x0,=20 le_prev =3D 0x0 },=20 v_cache_src =3D { lh_first =3D 0x0 },=20 v_cache_dst =3D { tqh_first =3D 0xfffff801a260f4d0,=20 tqh_last =3D 0xfffff801a260f4f0 },=20 v_cache_dd =3D 0x0,=20 v_lock =3D { lock_object =3D { lo_name =3D 0xffffffff81acefb6 "zfs",=20 lo_flags =3D 117112832,=20 lo_data =3D 0,=20 lo_witness =3D 0x0 },=20 lk_lock =3D 1,=20 lk_exslpfail =3D 0,=20 lk_timo =3D 51,=20 lk_pri =3D 96 },=20 v_interlock =3D { lock_object =3D { lo_name =3D 0xffffffff80fd2f2d "vnode interlock",=20 lo_flags =3D 16973824,=20 lo_data =3D 0,=20 lo_witness =3D 0x0 },=20 mtx_lock =3D 4 },=20 v_vnlock =3D 0xfffff8010bc087c8,=20 v_actfreelist =3D { tqe_next =3D 0xfffff80079317000,=20 tqe_prev =3D 0xfffff80092a9e820 },=20 v_bufobj =3D { bo_lock =3D { lock_object =3D { lo_name =3D 0xffffffff80fd2f3d "bufobj interlock",=20 lo_flags =3D 86179840,=20 lo_data =3D 0,=20 lo_witness =3D 0x0 },=20 rw_lock =3D 1 },=20 bo_ops =3D 0xffffffff8149bff0,=20 bo_object =3D 0xfffff80135aaa500,=20 bo_synclist =3D { le_next =3D 0x0,=20 le_prev =3D 0x0 },=20 bo_private =3D 0xfffff8010bc08760,=20 __bo_vnode =3D 0xfffff8010bc08760,=20 bo_clean =3D { bv_hd =3D { tqh_first =3D 0x0,=20 tqh_last =3D 0xfffff8010bc08880 },=20 bv_root =3D { pt_root =3D 0 },=20 bv_cnt =3D 0 },=20 bo_dirty =3D { bv_hd =3D { tqh_first =3D 0x0,=20 tqh_last =3D 0xfffff8010bc088a0 },=20 bv_root =3D { pt_root =3D 0 },=20 bv_cnt =3D 0 },=20 bo_numoutput =3D 0,=20 bo_flag =3D 0,=20 bo_bsize =3D 131072 },=20 v_pollinfo =3D 0x0,=20 v_label =3D 0x0,=20 v_lockf =3D 0x0,=20 v_rl =3D { rl_waiters =3D { tqh_first =3D 0x0,=20 tqh_last =3D 0xfffff8010bc088e8 },=20 rl_currdep =3D 0x0 },=20 v_cstart =3D 0,=20 v_lasta =3D 0,=20 v_lastw =3D 0,=20 v_clen =3D 0,=20 v_holdcnt =3D 2,=20 v_usecount =3D 2,=20 v_iflag =3D 512,=20 v_vflag =3D 0,=20 v_writecount =3D 0,=20 v_hash =3D 17547399,=20 v_type =3D VREG } (kgdb) print vp->v_cache_dst->tqh_first->nc_name $14 =3D 0xfffff801a260f512 "puppet20160226-52827-x6ea8y" The file where things go wrong for us is: /tmp/puppet20160226-52827-x6ea8y Performing a cat on this file also hangs on vodead. Any relevant waiting channels: # ps -o lwp -laxwwwSH | awk '{ if ($10 !~ "^(-|ttyin|lockf|wait|select|kqread|tx\->)") print; }'=20 LWP UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT=20=20= =20=20=20=20=20=20 TIME COMMAND 100000 0 0 0 0 -16 0 0 285088 swapin DLs -=20=20= =20=20=20 0:41.09 [kernel/swapper] 100048 0 0 0 0 -92 0 0 285088 vtbslp DLs -=20=20= =20=20=20 0:00.00 [kernel/virtio_ballo] 100018 0 2 0 0 -16 0 0 16 crypto_w DL -=20=20= =20=20=20 0:00.00 [crypto] 100019 0 3 0 0 -16 0 0 16 crypto_r DL -=20=20= =20=20=20 0:00.00 [crypto returns] 100057 0 5 0 0 20 0 0 6176 arc_recl DL -=20=20= =20=20 13:07.13 [zfskern/arc_reclaim] 100058 0 5 0 0 -8 0 0 6176 l2arc_fe DL -=20=20= =20=20=20 0:40.44 [zfskern/l2arc_feed_] 102486 0 5 0 0 -8 0 0 6176 zio->io_ DL -=20=20= =20=20=20 8:24.66 [zfskern/txg_thread_] 100062 0 7 0 0 -16 0 0 32 psleep DL -=20=20= =20=20 17:00.73 [pagedaemon/pagedaem] 100068 0 7 0 0 -16 0 0 32 umarcl DL -=20=20= =20=20=20 0:00.00 [pagedaemon/uma] 100063 0 8 0 0 -16 0 0 16 psleep DL -=20=20= =20=20=20 0:00.00 [vmdaemon] 100064 0 9 0 0 155 0 0 16 pgzero DL -=20=20= =20=20=20 0:00.17 [pagezero] 100001 0 10 0 0 -16 0 0 16 audit_wo DL -=20=20= =20=20 14:36.35 [audit] 100065 0 16 0 0 -16 0 0 16 psleep DL -=20=20= =20=20=20 0:51.50 [bufdaemon] 100066 0 17 0 0 -16 0 0 16 vlruwt DL -=20=20= =20=20=20 1:27.70 [vnlru] 100067 0 18 0 0 16 0 0 16 syncer DL -=20=20= =20=20 63:11.82 [syncer] 100406 65534 921 1 0 52 0 71104 33604 uwait Is -=20=20= =20=20=20 0:00.00 /usr/local/bin/memcached -l 127.0.0.1 -m 1024 -p 11211 -C -d -u nob= ody -P /var/run/memcached/memcached.pid 100387 0 958 1 0 23 0 16612 2224 nanslp Is -=20=20= =20=20 97:24.14 /usr/sbin/cron -s 100394 0 1194 0 0 -16 0 0 16 pftm DL -=20=20= =20=20 30:48.46 [pf purge] 100388 0 1201 1 0 52 0 14700 2256 sbwait Is -=20=20= =20=20=20 0:00.00 pflogd: [priv] (pflogd) 100375 64 1202 1201 0 20 0 14700 2312 bpf S -=20=20= =20=20=20 5:36.59 pflogd: [running] -s 116 -i pflog0 -f /dev/null (pflogd) 100389 0 52827 1 0 52 0 213452 112968 vodead T -=20=20= =20=20=20 0:08.69 ruby21: puppet agent: applying configuration (ruby21) 100400 0 55604 0 0 -16 0 0 16 ftcl DL -=20=20= =20=20=20 0:00.00 [ftcleanup] 101076 0 58064 1 0 20 0 16588 2168 auditd Is -=20=20= =20=20=20 0:07.02 /usr/sbin/auditd 101349 0 37651 37601 0 20 0 12356 2132 vodead D+ 0-=20= =20=20=20 0:00.00 cat puppet20160226-52827-x6ea8y This is the first time we're seeing this. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Feb 26 12:54:38 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04B94AB545E for ; Fri, 26 Feb 2016 12:54:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E882D1F0B for ; Fri, 26 Feb 2016 12:54:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1QCsbGh002774 for ; Fri, 26 Feb 2016 12:54:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Fri, 26 Feb 2016 12:54:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: johan@300.nl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 12:54:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 --- Comment #16 from johans --- I have to reboot the machine now, so hope gives enough extra information (though I doubt it). This was all the relevant information I could come up with. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Feb 26 18:37:10 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D906FAB46D2 for ; Fri, 26 Feb 2016 18:37:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA0E91CCB for ; Fri, 26 Feb 2016 18:37:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u1QIbAQ1002894 for ; Fri, 26 Feb 2016 18:37:10 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 207464] Panic when destroying ZFS snapshot on boot filesystem Date: Fri, 26 Feb 2016 18:37:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dustinwenz@ebureau.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 18:37:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207464 --- Comment #3 from dustinwenz@ebureau.com --- In the interest of simplification, I can confirm the problem exists in stable/r296074 GENERIC, with no kernel modifications. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Feb 27 03:51:52 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81FF9AB2D51 for ; Sat, 27 Feb 2016 03:51:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 634BD1FED for ; Sat, 27 Feb 2016 03:51:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id 60B5BAB2D4F; Sat, 27 Feb 2016 03:51:52 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60403AB2D4E for ; Sat, 27 Feb 2016 03:51:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 03B791FEC for ; Sat, 27 Feb 2016 03:51:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:9SmprhyhWlXC/OnXCy+O+j09IxM/srCxBDY+r6Qd0e8VIJqq85mqBkHD//Il1AaPBtWErawYwLSL+4nbGkU+or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6anHS+4HYoFwnlMkItf6KuStGU0Zj8ib360qaQSjsLrQL1Wal1IhSyoFeZnegtqqwmFJwMzADUqGBDYeVcyDAgD1uSmxHh+pX4p8Y7oGwD884mouRaTK73N4kmRLpDRGAsKWw4zMrzqQTYSwaToHAbVyMfj0wbLRLC6UTAX5zy+g7zvel51SzSadfzRLs3XTmnx7psRwLljD8HcTUwpjKEwvdshb5W9Ury7yd0xJTZNdmY X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CsAgDvG9FW/61jaINehAwsQQa6SQENgWYXCoI8gmxKAoF1FAEBAQEBAQEBYyeCLYIUAQEBAwEBAQEgBCcgCwULAgEIDgoCAg0ZAgInAQkmAgQIBwQBHASHdggOr0GOVAEBAQEBAQEDAQEBAQEBARUEe4UXgXSCRoQPAQYBAQWDGIE6BY0rdIhrhVmCb4IyhEaHaYUthXKIVQIeAQFCggMZgWYeLgeHCQEIFx1+AQEB X-IronPort-AV: E=Sophos;i="5.22,506,1449550800"; d="scan'208";a="268040196" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 26 Feb 2016 22:51:44 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 5AE1315F56D; Fri, 26 Feb 2016 22:51:44 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id JD9IJ0vGQfWk; Fri, 26 Feb 2016 22:51:43 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 2D5A715F56E; Fri, 26 Feb 2016 22:51:43 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id mqF-7Tp34eAt; Fri, 26 Feb 2016 22:51:43 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0CE4F15F56D; Fri, 26 Feb 2016 22:51:43 -0500 (EST) Date: Fri, 26 Feb 2016 22:51:43 -0500 (EST) From: Rick Macklem To: Bruce Evans Cc: fs@freebsd.org Message-ID: <1403082388.11082060.1456545103011.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20160226164613.N2180@besplex.bde.org> References: <20160226164613.N2180@besplex.bde.org> Subject: Re: silly write caching in nfs3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF44 (Win)/8.0.9_GA_6191) Thread-Topic: silly write caching in nfs3 Thread-Index: YIxK/xdpmoJvF6pX2tVxl52yQSHoEQ== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2016 03:51:52 -0000 Bruce Evans wrote: > nfs3 is slower than in old versions of FreeBSD. I debugged one of the > reasons today. > > Writes have apparently always done silly caching. Typical behaviour > is for iozone writing a 512MB file where the file fits in the buffer > cache/VMIO. The write is cached perfectly. But then when nfs_open() > reeopens the file, it calls vinvalbuf() to discard all of the cached > data. Thus nfs write caching usually discards useful older data to > make space for newer data that will never be never used (unless the > file is opened r/w and read using the same fd (and is not accessed > for a setattr or advlock operation -- these call vinvalbuf() too, if > NMODIFIED)). The discarding may be delayed for a long time. Then > keeping the useless data causes even more older data to be discarded. > Discarding it on close would at least prevent further loss. It would > have to be committed on close before discarding it of course. > Committing it on close does some good things even without discarding > there, and in oldnfs it gives a bug that prevents discaring in open -- > see below. > > nfs_open() does the discarding for different reasons in the NMODIFIED > and !NMODIFIED cases. In the NMODIFED case, it discard unconditionally. > This case can be avoided by fsync() before close or setting the sysctl > to commit in close. iozone does he fsync(). This helps in oldnfs but > not in newfs. With it, iozone on newfs now behaves like it did on oldnfs > 10-20 years ago. Something (perhaps just the timestamp bugs discussed > later) "fixed" the discarding on oldnfs 5-10 years ago. > > I think not committing in close is supposed to be an optimization, but > it is actually a pessimization for my kernel build tests (with object > files on nfs, which I normally avoid). Builds certainly have to reopen > files after writing them, to link them and perhaps to install them. > This causes the discarding. My kernel build tests also do a lot of > utimes() calls which cause the discarding before commit-on-close can > avoid the above cause for it it by clearing NMODIFIED. Enabling > commit-on-close gives a small optimisation with oldnfs by avoiding all > of the discarding except for utimes(). It reduces read RPCs by about > 25% without increasing write RPCs or real time. It decreases real time > by a few percent. > Well, the new NFS client code was cloned from the old one (about FreeBSD7). I did this so that the new client wouldn't exhibit different caching behaviour than the old one (avoiding any POLA). If you look in stable/10/sys/nfsclient/nfs_vnops.c and stable/10/sys/fs/nfsclient/nfs_clvnops.c at the nfs_open() and nfs_close() functions, the algorithm appears to be identical for NFSv3. (The new one has a bunch of NFSv4 gunk, but if you scratch out that stuff and ignore function name differences (nfs_flush() vs ncl_flush()), I think you'll find them the same. I couldn't spot any differences at a glance.) --> see r214513 in head/sys/fs/nfsclient/nfs_clvnops.c for example > The other reason for discarding is because the timestamps changed -- you > just wrote them, so the timestamps should have changed. Different bugs > in comparing the timestamps gave different misbehaviours. > > In old versions of FreeBSD and/or nfs, the timestamps had seconds > granularity, so many changes were missed. This explains mysterious > behaviours by iozone 10-20 years ago: the write caching is seen to > work perfectly for most small total sizes, since all the writes take > less than 1 second so the timestamps usually don't change (but sometimes > the writes lie across a seconds boundary so the timestamps do change). > > oldnfs was fixed many years ago to use timestamps with nanoseconds > resolution, but it doesn't suffer from the discarding in nfs_open() > in the !NMODIFIED case which is reached by either fsync() before close > of commit on close. I think this is because it updates n_mtime to > the server's new timestamp in nfs_writerpc(). This seems to be wrong, > since the file might have been written to by other clients and then > the change would not be noticed until much later if ever (setting the > timestamp prevents seeing it change when it is checked later, but you > might be able to see another metadata change). > > newfs has quite different code for nfs_writerpc(). Most of it was > moved to another function in nanother file. I understand this even > less, but it doesn't seem to have fetch the server's new timestamp or > update n_mtime in the v3 case. > I'm pretty sure it does capture the new attributes (including mtime in the reply. The function is called something like nfscl_loadattrcache(). In general, close-to-open consistency isn't needed for most mounts. (The only case where it matters is when multiple clients are concurrently updating files.) - There are a couple of options that might help performance when doing software builds on an NFS mount: nocto (I remember you don't like the name) - Actually, I can't remember why the code would still do the cache invalidation in nfs_open() when this is set. I wonder if the code in nfs_open() should maybe avoid invalidating the buffer cache when this is set? (I need to think about this.) noncontigwr - This one allows the writes to happen for byte aligned chunks when they are non-contiguous without pushing the individual writes to the server. (Again, this shouldn't cause problems unless multiple clients are writing to the file concurrently.) Both of these are worth trying for mounts where software builds are being done. > There are many other reasons why nfs is slower than in old versions. > One is that writes are more often done out of order. This tends to > give a slowness factor of about 2 unless the server can fix up the > order. I use an old server which can do the fixup for old clients but > not for newer clients starting in about FreeBSD-9 (or 7?). I actually thought this was mainly caused by the krpc that was introduced in FreeBSD7 (for both old and new NFS), separating the RPC from NFS. There are 2 layers in the krpc (sys/rpc/clnt_rc.c and sys/rpc/clnt_vc.c) that each use acquisition of a mutex to allow an RPC message to be sent. (Whichever thread happens to acquire the mutex first, sends first.) I had a couple of patches that tried to keep the RPC messages more ordered. (They would not have guaranteed exact ordering.) They seemed to help for the limited testing I could do, but since I wasn't seeing a lot of "out of order" reads/writes on my single core hardware, I couldn't verify how well these patches worked. mav@ was working on this at the time, but didn't get these patches tested either, from what I recall. --> Unfortunately, I seem to have lost these patches or I would have attached them so you could try them. Ouch. (I've cc'd mav@. Maybe he'll have them lying about. I think one was related to the nfsiod and the other for either sys/rpc/clnt_rc.c or sys/rpc/clnt_vc.c.) The patches were all client side. Maybe I'll try and recreate them. > I suspect > that this is just because Giant locking in old clients gave accidental > serialization. Multiple nfsiod's and/or nfsd's are are clearly needed > for performance if you have multiple NICs serving multiple mounts. Shared vnode locks are also a factor, at least for reads. (Before shared vnode locks, the vnode lock essentially serialized all reads.) As you note, a single threaded benchmark test is quite different than a lot of clients with a lot of threads doing I/O on a lot of files concurrently. The bandwidth * delay product of your network interconnect is also a factor. The larger this is, the more bits you need to be in transit to "fill the data pipe". You can increase the # of bits in transit by either using larger rsize/wsize or more read-ahead/write-behind. It would be nice to figure out why your case is performing better on the old nfs client (and/or server). If you have a fairly recent FreeBSD10 system, you could try doing mounts with new vs old client (and no other changes) and see what differences occur. (that would isolate new vs old from recent "old" and "really old") Good luck with it, rick > Other cases are less clear. For the iozone benchmark, there is only > 1 stream and multiple nfsiod's pessimize it into multiple streams that > give buffers which arrive out of order on the server if the multiple > nfsiod's are actually active. I use the following configuration to > ameliorate this, but the slowness factor is still often about 2 for > iozone: > - limit nfsd's to 4 > - limit nfsiod's to 4 > - limit nfs i/o sizes to 8K. The server fs block size is 16K, and > using a smaller block size usually helps by giving some delayed > writes which can be clustered better. (The non-nfs parts of the > server could be smarter and do this intentionally. The out-of-order > buffers look like random writes to the server.) 16K i/o sizes > otherwise work OK, but 32K i/o sizes are much slower for unknown > reasons. > > Bruce > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sat Feb 27 04:00:50 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36C90AB50A0 for ; Sat, 27 Feb 2016 04:00:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 18BA6365 for ; Sat, 27 Feb 2016 04:00:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id 17BF4AB509F; Sat, 27 Feb 2016 04:00:50 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 174A6AB509E for ; Sat, 27 Feb 2016 04:00:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id BE878364 for ; Sat, 27 Feb 2016 04:00:49 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:hl3bOxZJrRpt0c800IjSBln/LSx+4OfEezUN459isYplN5qZpcm5bnLW6fgltlLVR4KTs6sC0LqJ9f68EjJdqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh7/0pcGYPVgArQH+SI0xBS3+lR/WuMgSjNkqAYcK4TyNnEF1ff9Lz3hjP1OZkkW0zM6x+Jl+73YY4Kp5pIYTGZn9Ko4iULdVRBk4OmYurJnhrxXOZQyX+mYHVGgK1BFPBk7M8UepcI32t37At+F+kAyTNs7yQLV8DS6n5qxoTBLtoDoAOCM09HnXzMd52vEI6Cm9rgByltaHKLqeM+BzK/vQ X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CsAgBhHtFW/61jaINehAwsQQa6SQENgWYXCoUoSgKBdRQBAQEBAQEBAWMngi2CFAEBAQMBAQEBIAQnIAsFCwIBCA4KAgINGQICJwEJJgEBBAgHBAEcBId2CA6vOo5VAQEBAQEBAQMBAQEBAQEBFQR7hReBdIJGhBAGAQEFgxiBOgWOH4hrhVmCb4IyhEaHaYUthXKIVQIeAQFCggMZgWYeLgeHCggXHX4BAQE X-IronPort-AV: E=Sophos;i="5.22,506,1449550800"; d="scan'208";a="268040829" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 26 Feb 2016 23:00:48 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A44B315F56D; Fri, 26 Feb 2016 23:00:48 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 2N1DXjWeFFyG; Fri, 26 Feb 2016 23:00:47 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id BAAAD15F56E; Fri, 26 Feb 2016 23:00:47 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 33NgGMKy99q5; Fri, 26 Feb 2016 23:00:47 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9DC6A15F56D; Fri, 26 Feb 2016 23:00:47 -0500 (EST) Date: Fri, 26 Feb 2016 23:00:47 -0500 (EST) From: Rick Macklem To: Bruce Evans Cc: fs@freebsd.org Message-ID: <1347742231.11086226.1456545647628.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20160226164613.N2180@besplex.bde.org> References: <20160226164613.N2180@besplex.bde.org> Subject: Re: silly write caching in nfs3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF44 (Win)/8.0.9_GA_6191) Thread-Topic: silly write caching in nfs3 Thread-Index: AgtscdZHaat4mY+BTTXu5KB2CMN/Lw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2016 04:00:50 -0000 Bruce Evans wrote: > nfs3 is slower than in old versions of FreeBSD. I debugged one of the > reasons today. > > Writes have apparently always done silly caching. Typical behaviour > is for iozone writing a 512MB file where the file fits in the buffer > cache/VMIO. The write is cached perfectly. But then when nfs_open() > reeopens the file, it calls vinvalbuf() to discard all of the cached > data. Thus nfs write caching usually discards useful older data to > make space for newer data that will never be never used (unless the > file is opened r/w and read using the same fd (and is not accessed > for a setattr or advlock operation -- these call vinvalbuf() too, if > NMODIFIED)). The discarding may be delayed for a long time. Then > keeping the useless data causes even more older data to be discarded. > Discarding it on close would at least prevent further loss. It would > have to be committed on close before discarding it of course. > Committing it on close does some good things even without discarding > there, and in oldnfs it gives a bug that prevents discaring in open -- > see below. > > nfs_open() does the discarding for different reasons in the NMODIFIED > and !NMODIFIED cases. In the NMODIFED case, it discard unconditionally. > This case can be avoided by fsync() before close or setting the sysctl > to commit in close. iozone does he fsync(). This helps in oldnfs but > not in newfs. With it, iozone on newfs now behaves like it did on oldnfs > 10-20 years ago. Something (perhaps just the timestamp bugs discussed > later) "fixed" the discarding on oldnfs 5-10 years ago. > > I think not committing in close is supposed to be an optimization, but > it is actually a pessimization for my kernel build tests (with object > files on nfs, which I normally avoid). Builds certainly have to reopen > files after writing them, to link them and perhaps to install them. > This causes the discarding. My kernel build tests also do a lot of > utimes() calls which cause the discarding before commit-on-close can > avoid the above cause for it it by clearing NMODIFIED. Enabling > commit-on-close gives a small optimisation with oldnfs by avoiding all > of the discarding except for utimes(). It reduces read RPCs by about > 25% without increasing write RPCs or real time. It decreases real time > by a few percent. > > The other reason for discarding is because the timestamps changed -- you > just wrote them, so the timestamps should have changed. Different bugs > in comparing the timestamps gave different misbehaviours. > You could easily test to see if second-resolution timestamps make a difference by redefining the NFS_TIMESPEC_COMPARE() macro { in sys/fs/nfsclient/nfsnode.h } so that it only compares the tv_sec field and not the tv_nsec field. --> Then the client would only think the mtime has changed when tv_sec changes. rick > In old versions of FreeBSD and/or nfs, the timestamps had seconds > granularity, so many changes were missed. This explains mysterious > behaviours by iozone 10-20 years ago: the write caching is seen to > work perfectly for most small total sizes, since all the writes take > less than 1 second so the timestamps usually don't change (but sometimes > the writes lie across a seconds boundary so the timestamps do change). > > oldnfs was fixed many years ago to use timestamps with nanoseconds > resolution, but it doesn't suffer from the discarding in nfs_open() > in the !NMODIFIED case which is reached by either fsync() before close > of commit on close. I think this is because it updates n_mtime to > the server's new timestamp in nfs_writerpc(). This seems to be wrong, > since the file might have been written to by other clients and then > the change would not be noticed until much later if ever (setting the > timestamp prevents seeing it change when it is checked later, but you > might be able to see another metadata change). > > newfs has quite different code for nfs_writerpc(). Most of it was > moved to another function in nanother file. I understand this even > less, but it doesn't seem to have fetch the server's new timestamp or > update n_mtime in the v3 case. > > There are many other reasons why nfs is slower than in old versions. > One is that writes are more often done out of order. This tends to > give a slowness factor of about 2 unless the server can fix up the > order. I use an old server which can do the fixup for old clients but > not for newer clients starting in about FreeBSD-9 (or 7?). I suspect > that this is just because Giant locking in old clients gave accidental > serialization. Multiple nfsiod's and/or nfsd's are are clearly needed > for performance if you have multiple NICs serving multiple mounts. > Other cases are less clear. For the iozone benchmark, there is only > 1 stream and multiple nfsiod's pessimize it into multiple streams that > give buffers which arrive out of order on the server if the multiple > nfsiod's are actually active. I use the following configuration to > ameliorate this, but the slowness factor is still often about 2 for > iozone: > - limit nfsd's to 4 > - limit nfsiod's to 4 > - limit nfs i/o sizes to 8K. The server fs block size is 16K, and > using a smaller block size usually helps by giving some delayed > writes which can be clustered better. (The non-nfs parts of the > server could be smarter and do this intentionally. The out-of-order > buffers look like random writes to the server.) 16K i/o sizes > otherwise work OK, but 32K i/o sizes are much slower for unknown > reasons. > > Bruce > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sat Feb 27 04:21:15 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA03CAB5CCF for ; Sat, 27 Feb 2016 04:21:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B8CBAED7 for ; Sat, 27 Feb 2016 04:21:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id B801AAB5CCE; Sat, 27 Feb 2016 04:21:15 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7988AB5CCD for ; Sat, 27 Feb 2016 04:21:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 69A78ED6 for ; Sat, 27 Feb 2016 04:21:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id E733F1046FF0; Sat, 27 Feb 2016 15:21:04 +1100 (AEDT) Date: Sat, 27 Feb 2016 15:21:02 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans cc: fs@freebsd.org Subject: Re: silly write caching in nfs3 In-Reply-To: <20160226164613.N2180@besplex.bde.org> Message-ID: <20160227131353.V1337@besplex.bde.org> References: <20160226164613.N2180@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=EfU1O6SC c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=MMWvXMvT1jwRzmCVoGYA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2016 04:21:16 -0000 On Fri, 26 Feb 2016, Bruce Evans wrote: > nfs3 is slower than in old versions of FreeBSD. I debugged one of the > reasons today. > ... > oldnfs was fixed many years ago to use timestamps with nanoseconds > resolution, but it doesn't suffer from the discarding in nfs_open() > in the !NMODIFIED case which is reached by either fsync() before close > of commit on close. I think this is because it updates n_mtime to > the server's new timestamp in nfs_writerpc(). This seems to be wrong, > since the file might have been written to by other clients and then > the change would not be noticed until much later if ever (setting the > timestamp prevents seeing it change when it is checked later, but you > might be able to see another metadata change). > > newfs has quite different code for nfs_writerpc(). Most of it was > moved to another function in nanother file. I understand this even > less, but it doesn't seem to have fetch the server's new timestamp or > update n_mtime in the v3 case. This quick fix seems to give the same behaviour as in oldnfs. It also fixes some bugs in comments in nfs_fsync() (where I tried to pass a non-null cred, but none is available. The ARGSUSED bug is in many other functions): X Index: nfs_clvnops.c X =================================================================== X --- nfs_clvnops.c (revision 296089) X +++ nfs_clvnops.c (working copy) X @@ -1425,6 +1425,23 @@ X } X if (DOINGASYNC(vp)) X *iomode = NFSWRITE_FILESYNC; X + if (error == 0 && NFS_ISV3(vp)) { X + /* X + * Break seeing concurrent changes by other clients, X + * since without this the next nfs_open() would X + * invalidate our write buffers. This is worse than X + * useless unless the write is committed on close or X + * fsynced, since otherwise NMODIFIED remains set so X + * the next nfs_open() will still invalidate the write X + * buffers. Unfortunately, this cannot be placed in X + * ncl_flush() where NMODIFIED is cleared since X + * credentials are unavailable there for at least X + * calls by nfs_fsync(). X + */ X + mtx_lock(&(VTONFS(vp))->n_mtx); X + VTONFS(vp)->n_mtime = nfsva.na_mtime; X + mtx_unlock(&(VTONFS(vp))->n_mtx); X + } X if (error && NFS_ISV4(vp)) X error = nfscl_maperr(uiop->uio_td, error, (uid_t)0, (gid_t)0); X return (error); X @@ -2613,9 +2630,8 @@ X } X X /* X - * fsync vnode op. Just call ncl_flush() with commit == 1. X + * fsync vnode op. X */ X -/* ARGSUSED */ X static int X nfs_fsync(struct vop_fsync_args *ap) X { X @@ -2622,8 +2638,12 @@ X X if (ap->a_vp->v_type != VREG) { X /* X + * XXX: this comment is misformatted (after fixing its X + * internal errors) and misplaced. X + * X * For NFS, metadata is changed synchronously on the server, X - * so there is nothing to flush. Also, ncl_flush() clears X + * so the only thing to flush is data for regular files. X + * Also, ncl_flush() clears X * the NMODIFIED flag and that shouldn't be done here for X * directories. X */ > There are many other reasons why nfs is slower than in old versions. > One is that writes are more often done out of order. This tends to > give a slowness factor of about 2 unless the server can fix up the > order. I use an old server which can do the fixup for old clients but > not for newer clients starting in about FreeBSD-9 (or 7?). I suspect > that this is just because Giant locking in old clients gave accidental > serialization. Multiple nfsiod's and/or nfsd's are are clearly needed > for performance if you have multiple NICs serving multiple mounts. > Other cases are less clear. For the iozone benchmark, there is only > 1 stream and multiple nfsiod's pessimize it into multiple streams that > give buffers which arrive out of order on the server if the multiple > nfsiod's are actually active. I use the following configuration to > ameliorate this, but the slowness factor is still often about 2 for > iozone: > - limit nfsd's to 4 > - limit nfsiod's to 4 > - limit nfs i/o sizes to 8K. The server fs block size is 16K, and > using a smaller block size usually helps by giving some delayed > writes which can be clustered better. (The non-nfs parts of the > server could be smarter and do this intentionally. The out-of-order > buffers look like random writes to the server.) 16K i/o sizes > otherwise work OK, but 32K i/o sizes are much slower for unknown > reasons. Size 16K seems to work better now. I also use: - turn off most interrupt moderation. This reduces (ping) latency from ~125 usec to ~75 usec for em on PCIe (after already turning off interrupt moderation on the server to reduce it from 150-200 usec). 75 usec is still a lot, though it is about 3 times lower than the default misconfiguration. Downgrading up to older lem on PCI/33 reduces it to 52. Downgrading to DEVICE_POLLING reduces it to about 40. The dowgrades are upgrades :-(. Not using a switch reduces it by about another 20. Low latency important for small i/o's. I was suprised that it also helps a lot for large i/o's. Apparently it changes the timing enough to reduce the out-of-order buffers significantly. The default misconfiguration with 20 nfsiod's is worse than I expected (on an 8 core system). For (old) "iozone auto" which starts with a file size of 1MB, the write speed is about 2MB/sec with 20 nfsiod's and 22 MB/sec with 1 nfsiod. 2-4 nfsiod's work best. They give 30-40MB/sec for most file sizes. Apparently, with 20 nfsiod's the write of 1MB is split up into almost twenty pieces of 50K each (6 or 7 8K buffers each), and the final order is perhaps even worse than random. I think it is basically sequential with about seeks for all file sizes between 1MB and many MB. I also use: - no PREEMPTION and no IPI_PREEMPTION on SMP systems. This limits context switching. - no SCHED_ULE. HZ = 100. This also limits context switching. With more or fairer context switching, all nfsiods are more likely to run, causing more damage. More detailed result for iozone 1 65536 with nfsiodmax=64 and oldnfs and mostly best known other tuning: - first run write speed 2MB/S (probably still using 20) (all rates use disk marketing MB) - second run 9MB/S - after repeated runs, 250MB/S - the speed kept mostly dropping, and reached 21K/S - server stats for next run at 29K/S: 139 blocks tested and order of 24 fixed (the server has an early version of what is in -current, with more debugging) with nfsiodmax=20: - most runs 2-2.2MB/S; one at 750K/S - server stats for a run at 2.2MB/S: 135 blocks tested and 86 fixed with nfsiodmax=4: - 5.8-6.5MB/S - server stats for a run at 6.0MB/S: 135 blocks tested and 0 fixed with nfsiodmax=2: - 4.8-5.2MB/S - server stats for a run at 5.1MB/S: 138 blocks tested and 0 fixed with nfsiodmax=1: - 3.4MB/S - server stats: 138 blocks tested and 0 fixed For iozone 512 65536: with nfsiodmax=1: - 34.7MB/S - server stats: 65543 blocks tested and 0 fixed with nfsiodmax=2: - 45.9MB/S (this is close to the drive's speed and faster than direct on the server. It is faster because everything the clustering accidentally works better) - server stats: 65550 blocks tested and 578 fixed with nfsiodmax=4: - 45.6MB/S - server stats: 65550 blocks tested and 2067 fixed with nfsiodmax=20: - 21.4MB/S - server stats: 65576 blocks tested and 12057 fixed (it is easy to see how 7 nfsiods could give 1/7 = 14% of blocks out of order. The server is fixing up almost 20%, but that is not enough) with nfsiodmax=64 (caused server to not respond): - test aborted at 500+MB - server stats: about 10000 blocks fixed with nfsiodmax=64 again: - 9.6MB/S - server stats: 65598 blocks tested and 14034 fixed The nfsiod's get scheduled almost equally. Bruce From owner-freebsd-fs@freebsd.org Sat Feb 27 06:14:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00012AB62B0 for ; Sat, 27 Feb 2016 06:14:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E23281EDC for ; Sat, 27 Feb 2016 06:14:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id DEBB6AB62AF; Sat, 27 Feb 2016 06:14:39 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE3DBAB62AE for ; Sat, 27 Feb 2016 06:14:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail110.syd.optusnet.com.au (mail110.syd.optusnet.com.au [211.29.132.97]) by mx1.freebsd.org (Postfix) with ESMTP id 66E0D1EDB for ; Sat, 27 Feb 2016 06:14:38 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail110.syd.optusnet.com.au (Postfix) with ESMTPS id F4017782BA3; Sat, 27 Feb 2016 17:14:30 +1100 (AEDT) Date: Sat, 27 Feb 2016 17:14:29 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Rick Macklem cc: fs@freebsd.org Subject: Re: silly write caching in nfs3 In-Reply-To: <1403082388.11082060.1456545103011.JavaMail.zimbra@uoguelph.ca> Message-ID: <20160227153409.W1735@besplex.bde.org> References: <20160226164613.N2180@besplex.bde.org> <1403082388.11082060.1456545103011.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=EfU1O6SC c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=zJrO3aCF6v3l7-wk_r8A:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2016 06:14:40 -0000 On Fri, 26 Feb 2016, Rick Macklem wrote: > Bruce Evans wrote: >> nfs3 is slower than in old versions of FreeBSD. I debugged one of the >> reasons today. >> >> Writes have apparently always done silly caching. Typical behaviour >> is for iozone writing a 512MB file where the file fits in the buffer >> cache/VMIO. The write is cached perfectly. But then when nfs_open() >> reeopens the file, it calls vinvalbuf() to discard all of the cached >> data. Thus nfs write caching usually discards useful older data to >> ... >> I think not committing in close is supposed to be an optimization, but >> it is actually a pessimization for my kernel build tests (with object >> files on nfs, which I normally avoid). Builds certainly have to reopen >> files after writing them, to link them and perhaps to install them. >> This causes the discarding. My kernel build tests also do a lot of >> utimes() calls which cause the discarding before commit-on-close can >> avoid the above cause for it it by clearing NMODIFIED. Enabling >> commit-on-close gives a small optimisation with oldnfs by avoiding all >> of the discarding except for utimes(). It reduces read RPCs by about >> 25% without increasing write RPCs or real time. It decreases real time >> by a few percent. >> > Well, the new NFS client code was cloned from the old one (about FreeBSD7). > I did this so that the new client wouldn't exhibit different caching > behaviour than the old one (avoiding any POLA). > If you look in stable/10/sys/nfsclient/nfs_vnops.c and stable/10/sys/fs/nfsclient/nfs_clvnops.c > at the nfs_open() and nfs_close() functions, the algorithm appears to be > identical for NFSv3. (The new one has a bunch of NFSv4 gunk, but if you > scratch out that stuff and ignore function name differences (nfs_flush() vs > ncl_flush()), I think you'll find them the same. I couldn't spot any > differences at a glance.) > --> see r214513 in head/sys/fs/nfsclient/nfs_clvnops.c for example I blamed newnfs before :-), but when I looked at newnfs more closely I found that it was almost the same lexically in the most interesting places (but unfortunately has lexical differences from s/nfs/ncl/, and but doesn't have enough of these differences for debugging -- debugging is broken by having 2 static functions named nfs_foo() for many values of foo). But newnfs seems to have always been missing this critical code: X 1541 rgrimes int X 83651 peter nfs_writerpc(struct vnode *vp, struct uio *uiop, struct ucred *cred, X 158739 mohans int *iomode, int *must_commit) X 1541 rgrimes { X 9336 dfr if (v3) { X 9336 dfr wccflag = NFSV3_WCCCHK; X ... X 158739 mohans } X 158739 mohans if (wccflag) { X 158739 mohans mtx_lock(&(VTONFS(vp))->n_mtx); X 158739 mohans VTONFS(vp)->n_mtime = VTONFS(vp)->n_vattr.va_mtime; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ X 158739 mohans mtx_unlock(&(VTONFS(vp))->n_mtx); X 158739 mohans } This was in 4.4BSD-Lite1 under a slightly (?) different condition. BTW, how do you use svn to see the history of removed files? nfs_vnops.c has been removed in -current. I can find it in other branches, but is hard to find there even if you know where it is. This is no better than in cvs, where to find its full history I have cd know where it is in 3 different repositories that I have online and more that I should have. >> The other reason for discarding is because the timestamps changed -- you >> just wrote them, so the timestamps should have changed. Different bugs >> in comparing the timestamps gave different misbehaviours. >> >> In old versions of FreeBSD and/or nfs, the timestamps had seconds >> granularity, so many changes were missed. This explains mysterious >> behaviours by iozone 10-20 years ago: the write caching is seen to >> work perfectly for most small total sizes, since all the writes take >> less than 1 second so the timestamps usually don't change (but sometimes >> the writes lie across a seconds boundary so the timestamps do change). >> >> oldnfs was fixed many years ago to use timestamps with nanoseconds >> resolution, but it doesn't suffer from the discarding in nfs_open() >> in the !NMODIFIED case which is reached by either fsync() before close >> of commit on close. I think this is because it updates n_mtime to >> the server's new timestamp in nfs_writerpc(). This seems to be wrong, >> since the file might have been written to by other clients and then >> the change would not be noticed until much later if ever (setting the >> timestamp prevents seeing it change when it is checked later, but you >> might be able to see another metadata change). >> >> newfs has quite different code for nfs_writerpc(). Most of it was >> moved to another function in nanother file. I understand this even >> less, but it doesn't seem to have fetch the server's new timestamp or >> update n_mtime in the v3 case. >> > I'm pretty sure it does capture the new attributes (including mtime in > the reply. The function is called something like nfscl_loadattrcache(). Debugging shows that it loads the new attributes but doesn't clobber n_mtime with them. For a write test that takes 20 seconds, n_mtime sticks at its original value and the server time advances with each write by 20 seconds total (the server time only advances every second if the server timestamp precision is only 1 second). > In general, close-to-open consistency isn't needed for most mounts. > (The only case where it matters is when multiple clients are concurrently > updating files.) > - There are a couple of options that might help performance when doing > software builds on an NFS mount: > nocto (I remember you don't like the name) I actually do like it except for its negative logic. To turn it back on, you would need to use nonocto, but IIRC the negative logic for that is still broken (missing), so there is no way to turn it back on. > - Actually, I can't remember why the code would still do the cache > invalidation in nfs_open() when this is set. I wonder if the code > in nfs_open() should maybe avoid invalidating the buffer cache > when this is set? (I need to think about this.) I think it is technically correct for something to do the invalidation if NMODIFIED is still set in nfs_open(). nocto shouldn't and doesn't affect that. nocto is checked only in nfs_lookup() and only affects nfs_open() indirectly: its effect is that when nocto is not set, nfs_lookup() clears n_attrstamp which causes nfs_lookup() to do more, but hopefully still not cache invalidation. Cache invalidation is also done after a timeout and nocto doesn't affect that either. I still leave nocto off except for testing. I want to optimise the cto case, and my reference benchmarks are with cto. > noncontigwr - This one allows the writes to happen for byte aligned > chunks when they are non-contiguous without pushing the individual > writes to the server. (Again, this shouldn't cause problems unless > multiple clients are writing to the file concurrently.) > Both of these are worth trying for mounts where software builds are being > done. I tried this to see if it would fix the unordered writes. I didn't expect it to do much because I usually only have a single active client and a single active writer per file. It didn't make much difference. With nfsiods misordering writes, this option might give another source of silly writes. After it merges writes to give perfect contiguity, you send them to multiple nfsiods which might give perfect discontiguity (worse than random) :-). >> There are many other reasons why nfs is slower than in old versions. >> One is that writes are more often done out of order. This tends to >> give a slowness factor of about 2 unless the server can fix up the >> order. I use an old server which can do the fixup for old clients but >> not for newer clients starting in about FreeBSD-9 (or 7?). > I actually thought this was mainly caused by the krpc that was introduced > in FreeBSD7 (for both old and new NFS), separating the RPC from NFS. > There are 2 layers in the krpc (sys/rpc/clnt_rc.c and sys/rpc/clnt_vc.c) > that each use acquisition of a mutex to allow an RPC message to be sent. > (Whichever thread happens to acquire the mutex first, sends first.) I don't like the new krpc since it is larger and harder to debug (especially for me since I don't understand the old krpc either :-), but it is in FreeBSD-7 and in my main reference kernel r181717, and these don't have so many unordered blocks for at leasy writing. > I had a couple of patches that tried to keep the RPC messages more ordered. > (They would not have guaranteed exact ordering.) They seemed to help for > the limited testing I could do, but since I wasn't seeing a lot of > "out of order" reads/writes on my single core hardware, I couldn't verify I usually use single core hardware too, but saw some problems with 2 cores, and now with 8 cores the problems seem to be fundamental. > how well these patches worked. mav@ was working on this at the time, but > didn't get these patches tested either, from what I recall. > --> Unfortunately, I seem to have lost these patches or I would have > attached them so you could try them. Ouch. > (I've cc'd mav@. Maybe he'll have them lying about. I think one was > related to the nfsiod and the other for either sys/rpc/clnt_rc.c or > sys/rpc/clnt_vc.c.) > > The patches were all client side. Maybe I'll try and recreate them. It seems to require lots of communication between separate nfsiods to even preserve an order that has carefully been set up for them. If you have this then it is unclear why it can't be done more simply using a single nfsiod thread (per NIC or ifq). Only 1 thread should talk to the NIC/ifq, since you lose control if put other threads in between. If the NIC/ifq uses multiple threads then maintaining the order is its problem. >> I suspect >> that this is just because Giant locking in old clients gave accidental >> serialization. Multiple nfsiod's and/or nfsd's are are clearly needed >> for performance if you have multiple NICs serving multiple mounts. > Shared vnode locks are also a factor, at least for reads. > (Before shared vnode locks, the vnode lock essentially serialized all reads.) > > As you note, a single threaded benchmark test is quite different than a lot > of clients with a lot of threads doing I/O on a lot of files concurrently. It is also an important case for me. I mainly want creating of object files to be fast, and the cache invalidation and unordered blocks seem to be relatively even larger in this case. A typical compiler operation (if tmp or obj files are on nfs, which they shouldn't be but sometimes are) is: cc -pipe avoids creating intermediate file for preprocessor cc -c creates intermediate .S file for some compilers (not clang) .S file is written to cache reopening .S file to actually use it invalidates cache (workaround: (1) enable commit on close to clear NMODIFED, and (2) use 1-second timestamp resolution on server to break detection of the change, provided the file can be created and reopened without crossing a seconds boundary), or (3) use oldnfs cc -c creates intermediate .o file for all compilers similar considerations link step uses .o files and invalidates their cache in most cases (workaround: as above, except the whole compile usually takes more than 1 second, so the timestamp resolution hack doesn't work) install step similar considerations -- the linked file was the intermediate file for this step and reopening invalidates its cache. > The bandwidth * delay product of your network interconnect is also a factor. > The larger this is, the more bits you need to be in transit to "fill the data > pipe". You can increase the # of bits in transit by either using larger rsize/wsize > or more read-ahead/write-behind. I already have latency tuned to about 5-10 times smaller than on FreeBSD cluster machines, with the result that most operations are even more than 5-10 faster, due to smaller operations and most operations not having special support for keeping pipes full (if that is possible at all). > It would be nice to figure out why your case is performing better on the > old nfs client (and/or server). > > If you have a fairly recent FreeBSD10 system, you could try doing mounts > with new vs old client (and no other changes) and see what differences > occur. (that would isolate new vs old from recent "old" and "really old") Er, that is what I already did to isolate this problem. I have oldnfs and newfs in about 50 test kernels and finally isolated this problem in an up to date FreeBSD-10. Bruce From owner-freebsd-fs@freebsd.org Sat Feb 27 06:27:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADBB8AB656F for ; Sat, 27 Feb 2016 06:27:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9CA1E273 for ; Sat, 27 Feb 2016 06:27:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 96A5EAB656E; Sat, 27 Feb 2016 06:27:09 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 963EBAB656D for ; Sat, 27 Feb 2016 06:27:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail108.syd.optusnet.com.au (mail108.syd.optusnet.com.au [211.29.132.59]) by mx1.freebsd.org (Postfix) with ESMTP id 61527272 for ; Sat, 27 Feb 2016 06:27:09 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c110-21-41-193.carlnfd1.nsw.optusnet.com.au (c110-21-41-193.carlnfd1.nsw.optusnet.com.au [110.21.41.193]) by mail108.syd.optusnet.com.au (Postfix) with ESMTPS id 2E6581A19A9; Sat, 27 Feb 2016 17:27:00 +1100 (AEDT) Date: Sat, 27 Feb 2016 17:27:00 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Rick Macklem cc: fs@freebsd.org Subject: Re: silly write caching in nfs3 In-Reply-To: <1347742231.11086226.1456545647628.JavaMail.zimbra@uoguelph.ca> Message-ID: <20160227171437.N1735@besplex.bde.org> References: <20160226164613.N2180@besplex.bde.org> <1347742231.11086226.1456545647628.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=EfU1O6SC c=1 sm=1 tr=0 a=73JWPhLeruqQCjN69UNZtQ==:117 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=aJNDa8-r2z69CfcSr0sA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2016 06:27:09 -0000 On Fri, 26 Feb 2016, Rick Macklem wrote: > Bruce Evans wrote: >> nfs3 is slower than in old versions of FreeBSD. I debugged one of the >> reasons today. >... >> nfs_open() does the discarding for different reasons in the NMODIFIED >> and !NMODIFIED cases. In the NMODIFED case, it discard unconditionally. >> This case can be avoided by fsync() before close or setting the sysctl >> to commit in close. iozone does he fsync(). This helps in oldnfs but >> not in newfs. With it, iozone on newfs now behaves like it did on oldnfs >> 10-20 years ago. Something (perhaps just the timestamp bugs discussed >> later) "fixed" the discarding on oldnfs 5-10 years ago. >... >> The other reason for discarding is because the timestamps changed -- you >> just wrote them, so the timestamps should have changed. Different bugs >> in comparing the timestamps gave different misbehaviours. >> > You could easily test to see if second-resolution timestamps make a > difference by redefining the NFS_TIMESPEC_COMPARE() macro > { in sys/fs/nfsclient/nfsnode.h } so that it only compares the > tv_sec field and not the tv_nsec field. > --> Then the client would only think the mtime has changed when tv_sec > changes. My server has vfs.timestamp_precision for other reasons, so I get this automatically. So the fixes to use NFS_TIMESTAMP_COMPARE() don't actually work. This together with enabling write commit gives a useful optimization for compiling (it reduces the build time for a FreeBSD-4 kernel from 4.5 seconds to 4.4 seconds. Very useful for compiling hundreds of these :-). Write commit clears NMODIFIED and the buggy timestamp compare then avoids most cache invalidations. BTW, I don't kike the NFS_TIMESTAMP_COMPARE() macro. FreeBSD has a better standard macro timespeccmp(). The names of these macros have different bugs, but timespeccmp() is easier to use since it is shorter and you can see the comparison operator in it and it is not limited to comparing for inequality. NFS_TIMESTAMP_COMPARE() looks like it compares for equality, but actually compares for inequality. Bruce