From nobody Wed Dec 15 16:39:28 2021 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 660CF18DC468; Wed, 15 Dec 2021 16:39:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JDgv50Cklz4j2D; Wed, 15 Dec 2021 16:39:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DAC0E1F942; Wed, 15 Dec 2021 16:39:28 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1BFGdSZb011997; Wed, 15 Dec 2021 16:39:28 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1BFGdS2v011996; Wed, 15 Dec 2021 16:39:28 GMT (envelope-from git) Date: Wed, 15 Dec 2021 16:39:28 GMT Message-Id: <202112151639.1BFGdS2v011996@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Rick Macklem Subject: git: 867c27c23a5c - main - nfscl: Change IO_APPEND writes to direct I/O List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: rmacklem X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 867c27c23a5c469b27611cf53cc2390b5a193fa5 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1639586369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FTxpO6I5fI/+8oFIyz35KOaX5MSifAIdaEmZvfs5xRs=; b=kn04Wv/7zJfpKxKrBgAvEZJhouzUoKS/VkhbqGOyoFFS6AEc8G6ScTJD0SPyJB09+4FBTU OByOESmW34GV8jfGq5Ov+u5kx6MVLedAu0hEBMudJRra6wiz0DdDS/kCvQ0l3iGjhS1dB6 nhU+TNNQgXV2Dc3mJFJmJoF+asCN9B8H06lQcj30/vUJxY0mcUqxPuG/63OXtVxULJ7y41 afGJeVbD2KFRf7qJF+3UtMN+2sBXyaVnQrWMXAAa9AfJlrZoS6IN8WLOwkt8I5S3A3GndT 4t0TVeGwS2XywL0ZNhty+ZEdgUGRS1Ubuw2omX3em8Ilz7gp2IoTlXVOccpaPg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1639586369; a=rsa-sha256; cv=none; b=ZuclUv0eQSXfvsPrrAlnSrXQ0ISzh6snX2qIDoO3/LPHoKcUhw7Qjx6LDVWtu99J9FX2jB XhXCg5jn3nxG2ogGIHshgBPUsOZomYILTgbzVR8GxVeebR4ZjlRFqAOz6hSLp5+CAjN/vm ccVn5z0riUDO+O2hPRPNk7jYAmNCQKEkZRyT5uR8ZVlqpQn9zv15mNWwiSfhXFXpIKt3um VFt3hpJQVrXCaq8BFfYh/XoEsqtY3sb6rgO78YYU2VaUnUcLHjfFGz0BPOGu/L2jEfXT3y t8OBJY6T88yGEiBXKXEmlQhFwq9rWewJFFzYjkjMNBPJAo8ukj0ssh5IQuknEg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by rmacklem: URL: https://cgit.FreeBSD.org/src/commit/?id=867c27c23a5c469b27611cf53cc2390b5a193fa5 commit 867c27c23a5c469b27611cf53cc2390b5a193fa5 Author: Rick Macklem AuthorDate: 2021-12-15 16:35:48 +0000 Commit: Rick Macklem CommitDate: 2021-12-15 16:35:48 +0000 nfscl: Change IO_APPEND writes to direct I/O IO_APPEND writes have always been very slow over NFS, due to the need to acquire an up to date file size after flushing all writes to the NFS server. This patch switches the IO_APPEND writes to use direct I/O, bypassing the buffer cache. As such, flushing of writes normally only occurs when the open(..O_APPEND..) is done. It does imply that all writes must be done synchronously and must be committed to stable storage on the file server (NFSWRITE_FILESYNC). For a simple test program that does 10,000 IO_APPEND writes in a loop, performance improved significantly with this patch. For a UFS exported file system, the test ran 12x faster. This drops to 3x faster when the open(2)/close(2) are done for each loop iteration. For a ZFS exported file system, the test ran 40% faster. The much smaller improvement may have been because the ZFS file system I tested against does not have a ZIL log and does have "sync" enabled. Note that IO_APPEND write performance is still much slower than when done on local file systems. Although this is a simple patch, it does result in a significant semantics change, so I have given it a large MFC time. Tested by: otis MFC after: 3 months --- sys/fs/nfsclient/nfs_clbio.c | 6 +++++- sys/fs/nfsclient/nfs_clvnops.c | 8 ++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/sys/fs/nfsclient/nfs_clbio.c b/sys/fs/nfsclient/nfs_clbio.c index 29bc66669dfb..c3339617acce 100644 --- a/sys/fs/nfsclient/nfs_clbio.c +++ b/sys/fs/nfsclient/nfs_clbio.c @@ -1001,8 +1001,12 @@ ncl_write(struct vop_write_args *ap) if (uio->uio_resid == 0) return (0); - if (newnfs_directio_enable && (ioflag & IO_DIRECT) && vp->v_type == VREG) + if (vp->v_type == VREG && ((newnfs_directio_enable && (ioflag & + IO_DIRECT)) || (ioflag & IO_APPEND))) { + if ((ioflag & IO_APPEND) != 0) + ioflag |= IO_SYNC; return nfs_directio_write(vp, uio, cred, ioflag); + } /* * Maybe this should be above the vnode op call, but so long as diff --git a/sys/fs/nfsclient/nfs_clvnops.c b/sys/fs/nfsclient/nfs_clvnops.c index ba2ba27b8fb6..db2ef26a77c6 100644 --- a/sys/fs/nfsclient/nfs_clvnops.c +++ b/sys/fs/nfsclient/nfs_clvnops.c @@ -709,8 +709,8 @@ nfs_open(struct vop_open_args *ap) /* * If the object has >= 1 O_DIRECT active opens, we disable caching. */ - if (newnfs_directio_enable && (fmode & O_DIRECT) && - (vp->v_type == VREG)) { + if (vp->v_type == VREG && ((newnfs_directio_enable && (fmode & + O_DIRECT)) || (fmode & O_APPEND))) { if (np->n_directio_opens == 0) { NFSUNLOCKNODE(np); if (VOP_ISLOCKED(vp) != LK_EXCLUSIVE) { @@ -967,11 +967,11 @@ nfs_close(struct vop_close_args *ap) error = nfscl_maperr(ap->a_td, error, (uid_t)0, (gid_t)0); } - if (newnfs_directio_enable) + if (vp->v_type == VREG && ((newnfs_directio_enable && (fmode & + O_DIRECT)) || (fmode & O_APPEND))) { KASSERT((np->n_directio_asyncwr == 0), ("nfs_close: dirty unflushed (%d) directio buffers\n", np->n_directio_asyncwr)); - if (newnfs_directio_enable && (fmode & O_DIRECT) && (vp->v_type == VREG)) { NFSLOCKNODE(np); KASSERT((np->n_directio_opens > 0), ("nfs_close: unexpectedly value (0) of n_directio_opens\n"));