From owner-svn-src-projects@freebsd.org Tue Jul 4 23:04:31 2017 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 484E3D9CC40 for ; Tue, 4 Jul 2017 23:04:31 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C1A166413; Tue, 4 Jul 2017 23:04:31 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v64N4Uuc059329; Tue, 4 Jul 2017 23:04:30 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v64N4U2p059327; Tue, 4 Jul 2017 23:04:30 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201707042304.v64N4U2p059327@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Tue, 4 Jul 2017 23:04:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r320661 - projects/pnfs-planb-server-stable11/usr.sbin/nfsd X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/pnfs-planb-server-stable11/usr.sbin/nfsd X-SVN-Commit-Revision: 320661 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2017 23:04:31 -0000 Author: rmacklem Date: Tue Jul 4 23:04:29 2017 New Revision: 320661 URL: https://svnweb.freebsd.org/changeset/base/320661 Log: Add a pnfs.4 man page. Added: projects/pnfs-planb-server-stable11/usr.sbin/nfsd/pnfs.4 (contents, props changed) Modified: projects/pnfs-planb-server-stable11/usr.sbin/nfsd/Makefile Modified: projects/pnfs-planb-server-stable11/usr.sbin/nfsd/Makefile ============================================================================== --- projects/pnfs-planb-server-stable11/usr.sbin/nfsd/Makefile Tue Jul 4 23:02:12 2017 (r320660) +++ projects/pnfs-planb-server-stable11/usr.sbin/nfsd/Makefile Tue Jul 4 23:04:29 2017 (r320661) @@ -2,6 +2,6 @@ # $FreeBSD$ PROG= nfsd -MAN= nfsd.8 nfsv4.4 stablerestart.5 +MAN= nfsd.8 nfsv4.4 stablerestart.5 pnfs.4 .include Added: projects/pnfs-planb-server-stable11/usr.sbin/nfsd/pnfs.4 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ projects/pnfs-planb-server-stable11/usr.sbin/nfsd/pnfs.4 Tue Jul 4 23:04:29 2017 (r320661) @@ -0,0 +1,165 @@ +.\" Copyright (c) 2017 Rick Macklem +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd July 2, 2017 +.Dt PNFS 4 +.Os +.Sh NAME +.Nm pNFS +.Nd NFS Version 4.1 Parallel NFS Protocol +.Sh DESCRIPTION +The NFSv4.1 client and server provides support for the +.Tn pNFS +specification; see +.%T "Network File System (NFS) Version 4 Minor Version 1 Protocol RFC 5661" . +A pNFS service separates the Read/Write operations from all other NFSv4.1 +operations, which are referred to as Metadata operations. +The Read/Write operations are performed directly on the Data Server (DS) +where the file's data resides, bypassing the NFS server. +All other operations are performed on the NFS server, which is referred to +as a Metadata Server (MDS). +NFS clients that do not support +.Tn pNFS +perform Read/Write operations on the MDS, which acts as a proxy for the +appropriate DS. +.Pp +A FreeBSD pNFS service consists of a single MDS server plus one or more +DS servers, all of which are FreeBSD systems. +.Tn pNFS +clients mount the MDS as they would a single NFS server. +.Pp +A +.Tn pNFS +client must be running the +.Xr nfscbd 8 +daemon and use the mount options ''nfsv4,minorversion=1,pnfs''. +.Pp +A pNFS DS server must be configured as a NFSv4.1 server, where there is an +exported directory with subdirectories named ds0, ds1, ..., ds created +in it. +For example, if the exported directory is /ds and the number of subdirectories is 20, the subdirectories are named +/ds/ds0, /ds/ds1, ..., /ds/ds19. +This exported directory is the one that the MDS will mount via NFSv4.1 to use as +a DS. +The subdirectories are created simply to reduce the size of the directories +by spreading the data storage files across them. +If the +.Tn pNFS +service will be storing a large number of files, the service should be +configured with a large number of subdirectories. +There really is no disadvantage in having a large number of subdirectories, +so sysadmins should err on the side of creating many of them. +Each of these subdirectories must be owned by the that the -maproot +.Xr exports 5 +option maps to, since the MDS accesses these directories as . +These directories should have file mode 0700, so that only the mapped +for has access to them. +See +.Xr exports 5 +for more information on this. +These subdirectories must be created by the sysadmin on all DS servers before +the +.Tn pNFS +service is started. +.Pp +The sysctl +.sp +.Bd -literal -offset indent +vfs.nfsd.dsdirsize +.Ed +.Pp +defines the number of subdirectories named ds0, ds1, ... , ds, where N is +vfs.nfsd.dsdirsize - 1, with the default set to 20. +The number of subdirectories can be increased after the server has been +running, but only when the +.Xr nfsd 8 +daemon is not running. +.Pp +When files are created, the MDS creates a file tree identical to what a +single NFS server creates, except that all the regular (VREG) files will +be empty. +As such, if you look at the exported tree on the MDS directly +on the MDS server (not via an NFS mount), the files will all be of size zero. +Each of these files will also have two extended attributes in the system +attribute name space: +.sp +.Bd -literal -offset indent +pnfsd.dsfile - This extended attrbute stores the information that the + MDS needs to find the data storage file on a DS for this file. +pnfsd.dsattr - This extended attribute stores the Size, AccessTime, + ModifyTime and Change attributes for the file. +.Ed +.Pp +For each regular (VREG) file, the MDS creates a data storage file on one +of the DSs, in one of the ds subdirectories of the exported DS directory. +The name of this file is +the file handle of the file on the MDS in hexadecimal at time of creation. +.Pp +For +.Tn pNFS +clients, the service generates File Layout layouts and associated DeviceInfo. +For NFS clients that do not support NFSv4.1 pNFS, there will be a performance +hit, since the I/O RPCs will be proxied by the MDS for the DS server the +data storage file resides on. +.Pp +Configuration of a DS is done exactly as any other NFS server is configured, +with the data storage directory exported to the MDS. +.Pp +The MDS is configured to mount the data storage directories of the DSs. +For example, if there are 2 DSs named nfsv4-ds0 and nfsv4-ds1 and both of +these have a /ds directory exported to the MDS, the +.Xr fstab 5 +entries might be: +.sp +.Bd -literal -offset indent +nfsv4-ds0:/ds /ds0 nfs rw,nfsv4,minorversion=1 0 0 +nfsv4-ds1:/ds /ds1 nfs rw,nfsv4,minorversion=1 0 0 +.Ed +.Pp +The MDS will require a "-p" flag option specifying the DSs. For the above +mounts, the nfs_server_flags entry in +.Xr rc.conf 5 +might be: +.sp +.Bd -literal -offset indent +nfs_server_flags="-u -t -n 256 -p nfsv4-ds0:/ds0,nfsv4-ds1:/ds1" +.Ed +.Pp +Note that the "-p" flag requires the "mounted-on" directory path on the MDS. +.Sh SEE ALSO +.Xr pnfsdsfile 1 , +.Xr nfsv4 4 , +.Xr exports 5 , +.Xr fstab 5 , +.Xr rc.conf 5 , +.Xr nfscbd 8 , +.Xr nfsd 8 , +.Xr nfsuserd 8 +.Sh BUGS +At this time, there is no support for DS mirroring. +As such, the MDS plus all DS servers are single points of failure for the +.Tn pNFS +service.