From owner-freebsd-bugs@freebsd.org Thu Feb 27 07:42:58 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2C89B25808F for ; Thu, 27 Feb 2020 07:42:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48Sl5G0N7tz4GCY for ; Thu, 27 Feb 2020 07:42:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 0877C25808E; Thu, 27 Feb 2020 07:42:58 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 082F725808D for ; Thu, 27 Feb 2020 07:42:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Sl5F4C6dz4GBh for ; Thu, 27 Feb 2020 07:42:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 55EBCD5C1 for ; Thu, 27 Feb 2020 07:42:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 01R7gviN024032 for ; Thu, 27 Feb 2020 07:42:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 01R7gv5D024031 for bugs@FreeBSD.org; Thu, 27 Feb 2020 07:42:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 244465] ZFS recursive snapshot with refquota-full filesystems causes DoS for NFS users Date: Thu, 27 Feb 2020 07:42:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pen@lysator.liu.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 07:42:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D244465 Bug ID: 244465 Summary: ZFS recursive snapshot with refquota-full filesystems causes DoS for NFS users Product: Base System Version: 11.3-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: misc Assignee: bugs@FreeBSD.org Reporter: pen@lysator.liu.se This is probably related to the ZFS feature of throttling down the transact= ion size when writing data to very near-full (refquota) filesystems but anyway.= =20 System: Servers with many ZFS filesystems (HOME directories) for many users, shared= via NFSv4 and SMB (Samba). We are seeing DoS issues for NFS users on the same machine where some _othe= r_ user have filled their HOME directories so they are at, or very near (MB's), their refquota limits at the time when we are taking our hourly recursive snapshots of all filesystems on those servers. ("zfs snapshot -r DATA/homes" basically) The NFS users (they have their HOME directories mounted from those servers)= at those times are seeing freezes for over a minute where the NFS server basic= ally seems to be unresponsive. We are also seeing NFS mount requests beeing deni= ed (or rather timed out) at the same times. We haven't seen the same type of bug reports from our SMB users, but their usage patterns are a bit different so this might just be that they haven't noticed the same issue. Possible workarounds (not tested yet): 1. Don't use "zfs snapshot -r" but instead manually loop over all filesyste= ms and skip taking snapshots of the ones that are nearly full. (Modify the sou= rce for the "zfs" command to have an option to skip nearly-full filesystems when doing a recursive snap) 2. Temporarily increase the refquota of problematic filesystems before runn= ing "zfs snapshot -r". (Problem: We might not be able to lower the quota afterwards). 3. Modify the zfs snapshot stuff in the kernel to ignore the quotas (since = it's run by the root users I think if might be reasonable, but possibly not so e= asy to implement). In general this slowing down of everything ZFS-related when a filesystem is nearly full is starting to _really_ become a pain in the... 3. --=20 You are receiving this mail because: You are the assignee for the bug.=