From owner-svn-src-head@FreeBSD.ORG Mon May 4 15:23:59 2009 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 277E3106564A; Mon, 4 May 2009 15:23:59 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 136CD8FC17; Mon, 4 May 2009 15:23:59 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id n44FNxFi048797; Mon, 4 May 2009 15:23:59 GMT (envelope-from rmacklem@svn.freebsd.org) Received: (from rmacklem@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id n44FNxdS048794; Mon, 4 May 2009 15:23:59 GMT (envelope-from rmacklem@svn.freebsd.org) Message-Id: <200905041523.n44FNxdS048794@svn.freebsd.org> From: Rick Macklem Date: Mon, 4 May 2009 15:23:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r191783 - in head/sys/fs: nfs nfsclient nfsserver X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 May 2009 15:24:00 -0000 Author: rmacklem Date: Mon May 4 15:23:58 2009 New Revision: 191783 URL: http://svn.freebsd.org/changeset/base/191783 Log: Add the experimental nfs subtree to the kernel, that includes support for NFSv4 as well as NFSv2 and 3. It lives in 3 subdirs under sys/fs: nfs - functions that are common to the client and server nfsclient - a mutation of sys/nfsclient that call generic functions to do RPCs and handle state. As such, it retains the buffer cache handling characteristics and vnode semantics that are found in sys/nfsclient, for the most part. nfsserver - the server. It includes a DRC designed specifically for NFSv4, that is used instead of the generic DRC in sys/rpc. The build glue will be checked in later, so at this point, it consists of 3 new subdirs that should not affect kernel building. Approved by: kib (mentor) Added: head/sys/fs/nfs/ head/sys/fs/nfs/nfs.h (contents, props changed) head/sys/fs/nfs/nfs_commonacl.c (contents, props changed) head/sys/fs/nfs/nfs_commonkrpc.c (contents, props changed) head/sys/fs/nfs/nfs_commonport.c (contents, props changed) head/sys/fs/nfs/nfs_commonsubs.c (contents, props changed) head/sys/fs/nfs/nfs_var.h (contents, props changed) head/sys/fs/nfs/nfscl.h (contents, props changed) head/sys/fs/nfs/nfsclstate.h (contents, props changed) head/sys/fs/nfs/nfsdport.h (contents, props changed) head/sys/fs/nfs/nfskpiport.h (contents, props changed) head/sys/fs/nfs/nfsm_subs.h (contents, props changed) head/sys/fs/nfs/nfsport.h (contents, props changed) head/sys/fs/nfs/nfsproto.h (contents, props changed) head/sys/fs/nfs/nfsrvcache.h (contents, props changed) head/sys/fs/nfs/nfsrvstate.h (contents, props changed) head/sys/fs/nfs/nfsv4_errstr.h (contents, props changed) head/sys/fs/nfs/rpcv2.h (contents, props changed) head/sys/fs/nfs/xdr_subs.h (contents, props changed) head/sys/fs/nfsclient/ head/sys/fs/nfsclient/nfs.h (contents, props changed) head/sys/fs/nfsclient/nfs_clbio.c (contents, props changed) head/sys/fs/nfsclient/nfs_clcomsubs.c (contents, props changed) head/sys/fs/nfsclient/nfs_clkrpc.c (contents, props changed) head/sys/fs/nfsclient/nfs_cllock.c (contents, props changed) head/sys/fs/nfsclient/nfs_clnfsiod.c (contents, props changed) head/sys/fs/nfsclient/nfs_clnode.c (contents, props changed) head/sys/fs/nfsclient/nfs_clport.c (contents, props changed) head/sys/fs/nfsclient/nfs_clrpcops.c (contents, props changed) head/sys/fs/nfsclient/nfs_clstate.c (contents, props changed) head/sys/fs/nfsclient/nfs_clsubs.c (contents, props changed) head/sys/fs/nfsclient/nfs_clvfsops.c (contents, props changed) head/sys/fs/nfsclient/nfs_clvnops.c (contents, props changed) head/sys/fs/nfsclient/nfs_lock.h (contents, props changed) head/sys/fs/nfsclient/nfsargs.h (contents, props changed) head/sys/fs/nfsclient/nfsdiskless.h (contents, props changed) head/sys/fs/nfsclient/nfsmount.h (contents, props changed) head/sys/fs/nfsclient/nfsnode.h (contents, props changed) head/sys/fs/nfsclient/nlminfo.h (contents, props changed) head/sys/fs/nfsserver/ head/sys/fs/nfsserver/nfs_nfsdcache.c (contents, props changed) head/sys/fs/nfsserver/nfs_nfsdkrpc.c (contents, props changed) head/sys/fs/nfsserver/nfs_nfsdport.c (contents, props changed) head/sys/fs/nfsserver/nfs_nfsdserv.c (contents, props changed) head/sys/fs/nfsserver/nfs_nfsdsocket.c (contents, props changed) head/sys/fs/nfsserver/nfs_nfsdstate.c (contents, props changed) head/sys/fs/nfsserver/nfs_nfsdsubs.c (contents, props changed) Added: head/sys/fs/nfs/nfs.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/fs/nfs/nfs.h Mon May 4 15:23:58 2009 (r191783) @@ -0,0 +1,704 @@ +/*- + * Copyright (c) 1989, 1993 + * The Regents of the University of California. All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * Rick Macklem at The University of Guelph. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _NFS_NFS_H_ +#define _NFS_NFS_H_ +/* + * Tunable constants for nfs + */ + +#define NFS_MAXIOVEC 34 +#define NFS_TICKINTVL 10 /* Desired time for a tick (msec) */ +#define NFS_HZ (hz / nfscl_ticks) /* Ticks/sec */ +#define NFS_TIMEO (1 * NFS_HZ) /* Default timeout = 1 second */ +#define NFS_MINTIMEO (1 * NFS_HZ) /* Min timeout to use */ +#define NFS_MAXTIMEO (60 * NFS_HZ) /* Max timeout to backoff to */ +#define NFS_TCPTIMEO 300 /* TCP timeout */ +#define NFS_MAXRCVTIMEO 60 /* 1 minute in seconds */ +#define NFS_MINIDEMTIMEO (5 * NFS_HZ) /* Min timeout for non-idempotent ops*/ +#define NFS_MAXREXMIT 100 /* Stop counting after this many */ +#define NFSV4_CALLBACKTIMEO (2 * NFS_HZ) /* Timeout in ticks */ +#define NFSV4_CALLBACKRETRY 5 /* Number of retries before failure */ +#define NFSV4_CBRETRYCNT 4 /* # of CBRecall retries upon err */ +#define NFSV4_UPCALLTIMEO (15 * NFS_HZ) /* Timeout in ticks for upcalls */ + /* to gssd or nfsuserd */ +#define NFSV4_UPCALLRETRY 4 /* Number of retries before failure */ +#define NFS_MAXWINDOW 1024 /* Max number of outstanding requests */ +#define NFS_RETRANS 10 /* Num of retrans for soft mounts */ +#define NFS_MAXGRPS 16 /* Max. size of groups list */ +#define NFS_TRYLATERDEL 15 /* Maximum delay timeout (sec) */ +#ifndef NFS_REMOVETIMEO +#define NFS_REMOVETIMEO 15 /* # sec to wait for delegret in local syscall */ +#endif +#ifndef NFS_MINATTRTIMO +#define NFS_MINATTRTIMO 5 /* Attribute cache timeout in sec */ +#endif +#ifndef NFS_MAXATTRTIMO +#define NFS_MAXATTRTIMO 60 +#endif +#define NFS_WSIZE 8192 /* Def. write data size <= 8192 */ +#define NFS_RSIZE 8192 /* Def. read data size <= 8192 */ +#define NFS_READDIRSIZE 8192 /* Def. readdir size */ +#define NFS_DEFRAHEAD 0 /* Def. read ahead # blocks */ +#define NFS_MAXRAHEAD 32 /* Max. read ahead # blocks */ +#define NFS_MAXUIDHASH 64 /* Max. # of hashed uid entries/mp */ +#ifndef NFSRV_LEASE +#define NFSRV_LEASE 120 /* Lease time in seconds for V4 */ +#endif /* assigned to nfsrv_lease */ +#ifndef NFSRV_STALELEASE +#define NFSRV_STALELEASE (5 * nfsrv_lease) +#endif +#ifndef NFSRV_MOULDYLEASE +#define NFSRV_MOULDYLEASE 604800 /* One week (in sec) */ +#endif +#ifndef NFSCLIENTHASHSIZE +#define NFSCLIENTHASHSIZE 20 /* Size of server client hash table */ +#endif +#ifndef NFSLOCKHASHSIZE +#define NFSLOCKHASHSIZE 20 /* Size of server nfslock hash table */ +#endif +#define NFSSTATEHASHSIZE 10 /* Size of server stateid hash table */ +#ifndef NFSUSERHASHSIZE +#define NFSUSERHASHSIZE 30 /* Size of user id hash table */ +#endif +#ifndef NFSGROUPHASHSIZE +#define NFSGROUPHASHSIZE 5 /* Size of group id hash table */ +#endif +#ifndef NFSCLDELEGHIGHWATER +#define NFSCLDELEGHIGHWATER 10000 /* limit for client delegations */ +#endif +#ifndef NFSNOOPEN /* Inactive open owner (sec) */ +#define NFSNOOPEN 120 +#endif +#define NFSRV_LEASEDELTA 15 /* # of seconds to delay beyond lease */ +#define NFS_IDMAXSIZE 4 /* max sizeof (in_addr_t) */ +#ifndef NFSRVCACHE_UDPTIMEOUT +#define NFSRVCACHE_UDPTIMEOUT 30 /* # of sec to hold cached rpcs(udp) */ +#endif +#ifndef NFSRVCACHE_UDPHIGHWATER +#define NFSRVCACHE_UDPHIGHWATER 500 /* Max # of udp cache entries */ +#endif +#ifndef NFSRVCACHE_TCPTIMEOUT +#define NFSRVCACHE_TCPTIMEOUT (3600*12) /*#of sec to hold cached rpcs(tcp) */ +#endif +#ifndef NFSRVCACHE_FLOODLEVEL +#define NFSRVCACHE_FLOODLEVEL 16384 /* Very high water mark for cache */ +#endif +#ifndef NFSRV_CLIENTHIGHWATER +#define NFSRV_CLIENTHIGHWATER 1000 +#endif +#ifndef NFSRV_MAXDUMPLIST +#define NFSRV_MAXDUMPLIST 10000 +#endif +#ifndef NFS_ACCESSCACHESIZE +#define NFS_ACCESSCACHESIZE 8 +#endif +#define NFSV4_CBPORT 7745 /* Callback port for testing */ + +/* + * This macro defines the high water mark for issuing V4 delegations. + * (It is currently set at a conservative 20% of NFSRV_V4STATELIMIT. This + * may want to increase when clients can make more effective use of + * delegations.) + */ +#define NFSRV_V4DELEGLIMIT(c) (((c) * 5) > NFSRV_V4STATELIMIT) + +#define NFS_READDIRBLKSIZ DIRBLKSIZ /* Minimal nm_readdirsize */ + +/* + * Oddballs + */ +#define NFS_CMPFH(n, f, s) \ + ((n)->n_fhp->nfh_len == (s) && !NFSBCMP((n)->n_fhp->nfh_fh, (caddr_t)(f), (s))) +#define NFSRV_CMPFH(nf, ns, f, s) \ + ((ns) == (s) && !NFSBCMP((caddr_t)(nf), (caddr_t)(f), (s))) +#define NFS_CMPTIME(t1, t2) \ + ((t1).tv_sec == (t2).tv_sec && (t1).tv_nsec == (t2).tv_nsec) +#define NFS_SETTIME(t) do { \ + (t).tv_sec = time.tv_sec; (t).tv_nsec = 1000 * time.tv_usec; } while (0) +#define NFS_SRVMAXDATA(n) \ + (((n)->nd_flag & (ND_NFSV3 | ND_NFSV4)) ? \ + NFS_MAXDATA : NFS_V2MAXDATA) +#define NFS64BITSSET 0xffffffffffffffffull +#define NFS64BITSMINUS1 0xfffffffffffffffeull + +/* + * Structures for the nfssvc(2) syscall. Not that anyone but nfsd, mount_nfs + * and nfsloaduser should ever try and use it. + */ +struct nfsd_args { + int sock; /* Socket to serve */ + caddr_t name; /* Client addr for connection based sockets */ + int namelen; /* Length of name */ +}; + +/* + * nfsd argument for new krpc. + */ +struct nfsd_nfsd_args { + const char *principal; /* GSS-API service principal name */ + int minthreads; /* minimum service thread count */ + int maxthreads; /* maximum service thread count */ +}; + +/* + * Arguments for use by the callback daemon. + */ +struct nfsd_nfscbd_args { + const char *principal; /* GSS-API service principal name */ +}; + +struct nfscbd_args { + int sock; /* Socket to serve */ + caddr_t name; /* Client addr for connection based sockets */ + int namelen; /* Length of name */ + u_short port; /* Port# for callbacks */ +}; + +struct nfsd_idargs { + int nid_flag; /* Flags (see below) */ + uid_t nid_uid; /* user/group id */ + gid_t nid_gid; + int nid_usermax; /* Upper bound on user name cache */ + int nid_usertimeout;/* User name timeout (minutes) */ + u_char *nid_name; /* Name */ + int nid_namelen; /* and its length */ +}; + +struct nfsd_clid { + int nclid_idlen; /* Length of client id */ + u_char nclid_id[NFSV4_OPAQUELIMIT]; /* and name */ +}; + +struct nfsd_dumplist { + int ndl_size; /* Number of elements */ + void *ndl_list; /* and the list of elements */ +}; + +struct nfsd_dumpclients { + u_int32_t ndcl_flags; /* LCL_xxx flags */ + u_int32_t ndcl_nopenowners; /* Number of openowners */ + u_int32_t ndcl_nopens; /* and opens */ + u_int32_t ndcl_nlockowners; /* and of lockowners */ + u_int32_t ndcl_nlocks; /* and of locks */ + u_int32_t ndcl_ndelegs; /* and of delegations */ + u_int32_t ndcl_nolddelegs; /* and old delegations */ + sa_family_t ndcl_addrfam; /* Callback address */ + union { + struct in_addr sin_addr; + struct in6_addr sin6_addr; + } ndcl_cbaddr; + struct nfsd_clid ndcl_clid; /* and client id */ +}; + +struct nfsd_dumplocklist { + char *ndllck_fname; /* File Name */ + int ndllck_size; /* Number of elements */ + void *ndllck_list; /* and the list of elements */ +}; + +struct nfsd_dumplocks { + u_int32_t ndlck_flags; /* state flags NFSLCK_xxx */ + nfsv4stateid_t ndlck_stateid; /* stateid */ + u_int64_t ndlck_first; /* lock byte range */ + u_int64_t ndlck_end; + struct nfsd_clid ndlck_owner; /* Owner of open/lock */ + sa_family_t ndlck_addrfam; /* Callback address */ + union { + struct in_addr sin_addr; + struct in6_addr sin6_addr; + } ndlck_cbaddr; + struct nfsd_clid ndlck_clid; /* and client id */ +}; + +/* + * Structure for referral information. + */ +struct nfsreferral { + u_char *nfr_srvlist; /* List of servers */ + int nfr_srvcnt; /* number of servers */ + vnode_t nfr_vp; /* vnode for referral */ + u_int32_t nfr_dfileno; /* assigned dir inode# */ +}; + +/* + * Flags for lc_flags and opsflags for nfsrv_getclient(). + */ +#define LCL_NEEDSCONFIRM 0x00000001 +#define LCL_DONTCLEAN 0x00000002 +#define LCL_WAKEUPWANTED 0x00000004 +#define LCL_TCPCALLBACK 0x00000008 +#define LCL_CALLBACKSON 0x00000010 +#define LCL_INDEXNOTOK 0x00000020 +#define LCL_STAMPEDSTABLE 0x00000040 +#define LCL_EXPIREIT 0x00000080 +#define LCL_CBDOWN 0x00000100 +#define LCL_KERBV 0x00000400 +#define LCL_NAME 0x00000800 +#define LCL_NEEDSCBNULL 0x00001000 +#define LCL_GSSINTEGRITY 0x00002000 +#define LCL_GSSPRIVACY 0x00004000 +#define LCL_ADMINREVOKED 0x00008000 + +#define LCL_GSS LCL_KERBV /* Or of all mechs */ + +/* + * Bits for flags in nfslock and nfsstate. + * The access, deny, NFSLCK_READ and NFSLCK_WRITE bits must be defined as + * below, in the correct order, so the shifts work for tests. + */ +#define NFSLCK_READACCESS 0x00000001 +#define NFSLCK_WRITEACCESS 0x00000002 +#define NFSLCK_ACCESSBITS (NFSLCK_READACCESS | NFSLCK_WRITEACCESS) +#define NFSLCK_SHIFT 2 +#define NFSLCK_READDENY 0x00000004 +#define NFSLCK_WRITEDENY 0x00000008 +#define NFSLCK_DENYBITS (NFSLCK_READDENY | NFSLCK_WRITEDENY) +#define NFSLCK_SHAREBITS \ + (NFSLCK_READACCESS|NFSLCK_WRITEACCESS|NFSLCK_READDENY|NFSLCK_WRITEDENY) +#define NFSLCK_LOCKSHIFT 4 +#define NFSLCK_READ 0x00000010 +#define NFSLCK_WRITE 0x00000020 +#define NFSLCK_BLOCKING 0x00000040 +#define NFSLCK_RECLAIM 0x00000080 +#define NFSLCK_OPENTOLOCK 0x00000100 +#define NFSLCK_TEST 0x00000200 +#define NFSLCK_LOCK 0x00000400 +#define NFSLCK_UNLOCK 0x00000800 +#define NFSLCK_OPEN 0x00001000 +#define NFSLCK_CLOSE 0x00002000 +#define NFSLCK_CHECK 0x00004000 +#define NFSLCK_RELEASE 0x00008000 +#define NFSLCK_NEEDSCONFIRM 0x00010000 +#define NFSLCK_CONFIRM 0x00020000 +#define NFSLCK_DOWNGRADE 0x00040000 +#define NFSLCK_DELEGREAD 0x00080000 +#define NFSLCK_DELEGWRITE 0x00100000 +#define NFSLCK_DELEGCUR 0x00200000 +#define NFSLCK_DELEGPREV 0x00400000 +#define NFSLCK_OLDDELEG 0x00800000 +#define NFSLCK_DELEGRECALL 0x01000000 +#define NFSLCK_SETATTR 0x02000000 +#define NFSLCK_DELEGPURGE 0x04000000 +#define NFSLCK_DELEGRETURN 0x08000000 + +/* And bits for nid_flag */ +#define NFSID_INITIALIZE 0x0001 +#define NFSID_ADDUID 0x0002 +#define NFSID_DELUID 0x0004 +#define NFSID_ADDUSERNAME 0x0008 +#define NFSID_DELUSERNAME 0x0010 +#define NFSID_ADDGID 0x0020 +#define NFSID_DELGID 0x0040 +#define NFSID_ADDGROUPNAME 0x0080 +#define NFSID_DELGROUPNAME 0x0100 + +/* + * Stats structure + */ +struct nfsstats { + int attrcache_hits; + int attrcache_misses; + int lookupcache_hits; + int lookupcache_misses; + int direofcache_hits; + int direofcache_misses; + int accesscache_hits; + int accesscache_misses; + int biocache_reads; + int read_bios; + int read_physios; + int biocache_writes; + int write_bios; + int write_physios; + int biocache_readlinks; + int readlink_bios; + int biocache_readdirs; + int readdir_bios; + int rpccnt[NFS_NPROCS]; + int rpcretries; + int srvrpccnt[NFSV4OP_NOPS + NFSV4OP_FAKENOPS]; + int srvrpc_errs; + int srv_errs; + int rpcrequests; + int rpctimeouts; + int rpcunexpected; + int rpcinvalid; + int srvcache_inproghits; + int srvcache_idemdonehits; + int srvcache_nonidemdonehits; + int srvcache_misses; + int srvcache_tcppeak; + int srvcache_size; + int srvclients; + int srvopenowners; + int srvopens; + int srvlockowners; + int srvlocks; + int srvdelegates; + int cbrpccnt[NFSV4OP_CBNOPS]; + int clopenowners; + int clopens; + int cllockowners; + int cllocks; + int cldelegates; + int cllocalopenowners; + int cllocalopens; + int cllocallockowners; + int cllocallocks; +}; + +/* + * fs.nfs sysctl(3) identifiers + */ +#define NFS_NFSSTATS 1 /* struct: struct nfsstats */ + +#define FS_NFS_NAMES { \ + { 0, 0 }, \ + { "nfsstats", CTLTYPE_STRUCT }, \ +} + +/* + * Here is the definition of the attribute bits array and macros that + * manipulate it. + * THE MACROS MUST BE MANUALLY MODIFIED IF NFSATTRBIT_MAXWORDS CHANGES!! + * It is (NFSATTRBIT_MAX + 31) / 32. + */ +#define NFSATTRBIT_MAXWORDS 2 + +typedef struct { + u_int32_t bits[NFSATTRBIT_MAXWORDS]; +} nfsattrbit_t; + +#define NFSZERO_ATTRBIT(b) do { (b)->bits[0] = 0; (b)->bits[1] = 0; } while (0) +#define NFSSET_ATTRBIT(t, f) do { (t)->bits[0] = (f)->bits[0]; \ + (t)->bits[1] = (f)->bits[1]; } while (0) +#define NFSSETSUPP_ATTRBIT(b) do { \ + (b)->bits[0] = NFSATTRBIT_SUPP0; \ + (b)->bits[1] = (NFSATTRBIT_SUPP1 | NFSATTRBIT_SUPPSETONLY); } while (0) +#define NFSISSET_ATTRBIT(b, p) ((b)->bits[(p) / 32] & (1 << ((p) % 32))) +#define NFSSETBIT_ATTRBIT(b, p) ((b)->bits[(p) / 32] |= (1 << ((p) % 32))) +#define NFSCLRBIT_ATTRBIT(b, p) ((b)->bits[(p) / 32] &= ~(1 << ((p) % 32))) +#define NFSCLRALL_ATTRBIT(b, a) do { \ + (b)->bits[0] &= ~((a)->bits[0]); \ + (b)->bits[1] &= ~((a)->bits[1]); \ + } while (0) +#define NFSCLRNOT_ATTRBIT(b, a) do { \ + (b)->bits[0] &= ((a)->bits[0]); \ + (b)->bits[1] &= ((a)->bits[1]); \ + } while (0) +#define NFSCLRNOTFILLABLE_ATTRBIT(b) do { \ + (b)->bits[0] &= NFSATTRBIT_SUPP0; \ + (b)->bits[1] &= NFSATTRBIT_SUPP1; } while (0) +#define NFSCLRNOTSETABLE_ATTRBIT(b) do { \ + (b)->bits[0] &= NFSATTRBIT_SETABLE0; \ + (b)->bits[1] &= NFSATTRBIT_SETABLE1; } while (0) +#define NFSNONZERO_ATTRBIT(b) ((b)->bits[0] || (b)->bits[1]) +#define NFSEQUAL_ATTRBIT(b, p) \ + ((b)->bits[0] == (p)->bits[0] && (b)->bits[1] == (p)->bits[1]) +#define NFSGETATTR_ATTRBIT(b) do { \ + (b)->bits[0] = NFSATTRBIT_GETATTR0; \ + (b)->bits[1] = NFSATTRBIT_GETATTR1; } while (0) +#define NFSWCCATTR_ATTRBIT(b) do { \ + (b)->bits[0] = NFSATTRBIT_WCCATTR0; \ + (b)->bits[1] = NFSATTRBIT_WCCATTR1; } while (0) +#define NFSWRITEGETATTR_ATTRBIT(b) do { \ + (b)->bits[0] = NFSATTRBIT_WRITEGETATTR0; \ + (b)->bits[1] = NFSATTRBIT_WRITEGETATTR1; } while (0) +#define NFSCBGETATTR_ATTRBIT(b, c) do { \ + (c)->bits[0] = ((b)->bits[0] & NFSATTRBIT_CBGETATTR0); \ + (c)->bits[1] = ((b)->bits[1] & NFSATTRBIT_CBGETATTR1); } while (0) +#define NFSPATHCONF_GETATTRBIT(b) do { \ + (b)->bits[0] = NFSGETATTRBIT_PATHCONF0; \ + (b)->bits[1] = NFSGETATTRBIT_PATHCONF1; } while (0) +#define NFSSTATFS_GETATTRBIT(b) do { \ + (b)->bits[0] = NFSGETATTRBIT_STATFS0; \ + (b)->bits[1] = NFSGETATTRBIT_STATFS1; } while (0) +#define NFSISSETSTATFS_ATTRBIT(b) \ + (((b)->bits[0] & NFSATTRBIT_STATFS0) || \ + ((b)->bits[1] & NFSATTRBIT_STATFS1)) +#define NFSCLRSTATFS_ATTRBIT(b) do { \ + (b)->bits[0] &= ~NFSATTRBIT_STATFS0; \ + (b)->bits[1] &= ~NFSATTRBIT_STATFS1; } while (0) +#define NFSREADDIRPLUS_ATTRBIT(b) do { \ + (b)->bits[0] = NFSATTRBIT_READDIRPLUS0; \ + (b)->bits[1] = NFSATTRBIT_READDIRPLUS1; } while (0) +#define NFSREFERRAL_ATTRBIT(b) do { \ + (b)->bits[0] = NFSATTRBIT_REFERRAL0; \ + (b)->bits[1] = NFSATTRBIT_REFERRAL1; } while (0) + +/* + * Store uid, gid creds that handle maps to. + * Since some BSDen define cr_gid as cr_groups[0], I'll just keep them + * all in nfsc_groups[NGROUPS + 1]. + */ +struct nfscred { + uid_t nfsc_uid; + gid_t nfsc_groups[NGROUPS + 1]; + int nfsc_ngroups; +}; + +/* + * Constants that define the file handle for the V4 root directory. + * (The FSID must never be used by other file systems that are exported.) + */ +#define NFSV4ROOT_FSID0 ((int32_t) -1) +#define NFSV4ROOT_FSID1 ((int32_t) -1) +#define NFSV4ROOT_REFERRAL ((int32_t) -2) +#define NFSV4ROOT_INO 2 /* It's traditional */ +#define NFSV4ROOT_GEN 1 + +/* + * The set of signals the interrupt an I/O in progress for NFSMNT_INT mounts. + * What should be in this set is open to debate, but I believe that since + * I/O system calls on ufs are never interrupted by signals the set should + * be minimal. My reasoning is that many current programs that use signals + * such as SIGALRM will not expect file I/O system calls to be interrupted + * by them and break. + */ +#if defined(_KERNEL) || defined(KERNEL) + +struct uio; struct buf; struct vattr; struct nameidata; /* XXX */ + +/* + * Socket errors ignored for connectionless sockets? + * For now, ignore them all + */ +#define NFSIGNORE_SOERROR(s, e) \ + ((e) != EINTR && (e) != ERESTART && (e) != EWOULDBLOCK && \ + ((s) & PR_CONNREQUIRED) == 0) + + +/* + * This structure holds socket information for a connection. Used by the + * client and the server for callbacks. + */ +struct nfssockreq { + NFSSOCKADDR_T nr_nam; + int nr_sotype; + int nr_soproto; + int nr_soflags; + struct ucred *nr_cred; + int nr_lock; + NFSMUTEX_T nr_mtx; + u_int32_t nr_prog; + u_int32_t nr_vers; + struct __rpc_client *nr_client; +}; + +/* + * And associated nr_lock bits. + */ +#define NFSR_SNDLOCK 0x01 +#define NFSR_WANTSND 0x02 +#define NFSR_RCVLOCK 0x04 +#define NFSR_WANTRCV 0x08 +#define NFSR_RESERVEDPORT 0x10 +#define NFSR_LOCALHOST 0x20 + +/* + * Queue head for nfsreq's + */ +TAILQ_HEAD(nfsreqhead, nfsreq); + +/* First 8 R_xxx flags defined in rpc/rpcclnt.h, the rest are here */ +#define R_DONTRECOVER 0x00000100 /* don't initiate recovery when this + rpc gets a stale state reply */ + +/* + * Network address hash list element + */ +union nethostaddr { + struct in_addr had_inet; + struct in6_addr had_inet6; +}; + +/* + * Structure of list of mechanisms. + */ +struct nfsgss_mechlist { + int len; + const u_char *str; + int totlen; +}; +#define KERBV_MECH 0 /* position in list */ + +/* + * This structure is used by the server for describing each request. + */ +struct nfsrv_descript { + mbuf_t nd_mrep; /* Request mbuf list */ + mbuf_t nd_md; /* Current dissect mbuf */ + mbuf_t nd_mreq; /* Reply mbuf list */ + mbuf_t nd_mb; /* Current build mbuf */ + NFSSOCKADDR_T nd_nam; /* and socket addr */ + NFSSOCKADDR_T nd_nam2; /* return socket addr */ + caddr_t nd_dpos; /* Current dissect pos */ + caddr_t nd_bpos; /* Current build pos */ + u_int16_t nd_procnum; /* RPC # */ + u_int32_t nd_flag; /* nd_flag */ + u_int32_t nd_repstat; /* Reply status */ + int *nd_errp; /* Pointer to ret status */ + u_int32_t nd_retxid; /* Reply xid */ + struct nfsrvcache *nd_rp; /* Assoc. cache entry */ + struct timeval nd_starttime; /* Time RPC initiated */ + fhandle_t nd_fh; /* File handle */ + struct ucred *nd_cred; /* Credentials */ + uid_t nd_saveduid; /* Saved uid */ + u_int64_t nd_sockref; /* Rcv socket ref# */ + u_int64_t nd_compref; /* Compound RPC ref# */ + time_t nd_tcpconntime; /* Time TCP connection est. */ + nfsquad_t nd_clientid; /* Implied clientid */ + int nd_credflavor; /* credential flavor */ + int nd_gssnamelen; /* principal name length */ + char *nd_gssname; /* principal name */ +}; + +#define nd_princlen nd_gssnamelen +#define nd_principal nd_gssname + +/* Bits for "nd_flag" */ +#define ND_DONTSAVEREPLY 0x00000001 +#define ND_SAVEREPLY 0x00000002 +#define ND_NFSV2 0x00000004 +#define ND_NFSV3 0x00000008 +#define ND_NFSV4 0x00000010 +#define ND_KERBV 0x00000020 +#define ND_GSSINTEGRITY 0x00000040 +#define ND_GSSPRIVACY 0x00000080 +#define ND_WINDOWVERF 0x00000100 +#define ND_GSSINITREPLY 0x00000200 +#define ND_STREAMSOCK 0x00000400 +#define ND_PUBLOOKUP 0x00000800 +#define ND_USEGSSNAME 0x00001000 +#define ND_SAMETCPCONN 0x00002000 +#define ND_IMPLIEDCLID 0x00004000 +#define ND_NOMOREDATA 0x00008000 +#define ND_V4WCCATTR 0x00010000 +#define ND_NFSCB 0x00020000 +#define ND_AUTHNONE 0x00040000 +#define ND_EXGSSONLY 0x00080000 +#define ND_INCRSEQID 0x00100000 + +/* + * ND_GSS should be the "or" of all GSS type authentications. + */ +#define ND_GSS (ND_KERBV) + +struct nfsv4_opflag { + int retfh; + int needscfh; + int savereply; + int modifyfs; +}; + +/* + * Flags used to indicate what to do w.r.t. seqid checking. + */ +#define NFSRVSEQID_FIRST 0x01 +#define NFSRVSEQID_LAST 0x02 +#define NFSRVSEQID_OPEN 0x04 + +/* + * MNT_EXGSSONLY is the Or of all the EXGSS bits. + */ +#define MNT_EXGSSONLY MNT_EXGSSKRB5 + +/* + * assign a doubly linked list to a new head + * and prepend one list into another. + */ +#define LIST_NEWHEAD(nhead, ohead, field) do { \ + if (((nhead)->lh_first = (ohead)->lh_first) != NULL) \ + (ohead)->lh_first->field.le_prev = &(nhead)->lh_first; \ + (ohead)->lh_first = NULL; \ + } while (0) + +#define LIST_PREPEND(head, phead, lelm, field) do { \ + if ((head)->lh_first != NULL) { \ + (lelm)->field.le_next = (head)->lh_first; \ + (lelm)->field.le_next->field.le_prev = \ + &(lelm)->field.le_next; \ + } \ + (head)->lh_first = (phead)->lh_first; \ + (head)->lh_first->field.le_prev = &(head)->lh_first; \ + } while (0) + +/* + * File handle structure for client. Malloc'd to the correct length with + * malloc type M_NFSFH. + */ +struct nfsfh { + u_int16_t nfh_len; /* Length of file handle */ + u_int8_t nfh_fh[1]; /* and the file handle */ +}; + +/* + * File handle structure for server. The NFSRV_MAXFH constant is + * set in nfsdport.h. I use a 32bit length, so that alignment is + * preserved. + */ +struct nfsrvfh { + u_int32_t nfsrvfh_len; + u_int8_t nfsrvfh_data[NFSRV_MAXFH]; +}; + +/* + * This structure is used for sleep locks on the NFSv4 nfsd threads and + * NFSv4 client data structures. + */ +struct nfsv4lock { + u_int32_t nfslock_usecnt; + u_int8_t nfslock_lock; +}; +#define NFSV4LOCK_LOCK 0x01 +#define NFSV4LOCK_LOCKWANTED 0x02 +#define NFSV4LOCK_WANTED 0x04 + +/* + * Values for the override argument for nfsvno_accchk(). + */ +#define NFSACCCHK_NOOVERRIDE 0 +#define NFSACCCHK_ALLOWROOT 1 +#define NFSACCCHK_ALLOWOWNER 2 + +/* + * and values for the vpislocked argument for nfsvno_accchk(). + */ +#define NFSACCCHK_VPNOTLOCKED 0 +#define NFSACCCHK_VPISLOCKED 1 + +#endif /* _KERNEL */ + +#endif /* _NFS_NFS_H */ Added: head/sys/fs/nfs/nfs_commonacl.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/fs/nfs/nfs_commonacl.c Mon May 4 15:23:58 2009 (r191783) @@ -0,0 +1,750 @@ +/*- + * Copyright (c) 2009 Rick Macklem, University of Guelph + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#ifndef APPLEKEXT +#include + +extern int nfsrv_useacl; +#endif + +static int nfsrv_acemasktoperm(u_int32_t acetype, u_int32_t mask, int owner, + enum vtype type, acl_perm_t *permp); + +#if defined(NFS4_ACL_EXTATTR_NAME) +/* + * Handle xdr for an ace. + */ +APPLESTATIC int +nfsrv_dissectace(struct nfsrv_descript *nd, struct acl_entry *acep, + int *aceerrp, int *acesizep, NFSPROC_T *p) +{ + u_int32_t *tl; + int len, gotid = 0, owner = 0, error = 0, aceerr = 0; + u_char *name, namestr[NFSV4_SMALLSTR + 1]; + u_int32_t flag, mask, acetype; + gid_t gid; + uid_t uid; + + *aceerrp = 0; + acep->ae_flags = 0; + NFSM_DISSECT(tl, u_int32_t *, 4 * NFSX_UNSIGNED); + acetype = fxdr_unsigned(u_int32_t, *tl++); + flag = fxdr_unsigned(u_int32_t, *tl++); + mask = fxdr_unsigned(u_int32_t, *tl++); + len = fxdr_unsigned(int, *tl); + if (len < 0) { + return (NFSERR_BADXDR); + } else if (len == 0) { + /* Netapp filers return a 0 length who for nil users */ + acep->ae_tag = ACL_UNDEFINED_TAG; + acep->ae_id = ACL_UNDEFINED_ID; + acep->ae_perm = (acl_perm_t)0; + acep->ae_extended = ACL_EXTENDED_DENY; + if (acesizep) + *acesizep = 4 * NFSX_UNSIGNED; + return (0); + } + if (len > NFSV4_SMALLSTR) + name = malloc(len + 1, M_NFSSTRING, M_WAITOK); + else + name = namestr; + error = nfsrv_mtostr(nd, name, len); + if (error) { + if (len > NFSV4_SMALLSTR) + free(name, M_NFSSTRING); + return (error); + } + if (len == 6) { + if (!NFSBCMP(name, "OWNER@", 6)) { + acep->ae_tag = ACL_USER_OBJ; + acep->ae_id = ACL_UNDEFINED_ID; + owner = 1; + gotid = 1; + } else if (!NFSBCMP(name, "GROUP@", 6)) { + acep->ae_tag = ACL_GROUP_OBJ; + acep->ae_id = ACL_UNDEFINED_ID; + gotid = 1; + } + } else if (len == 9 && !NFSBCMP(name, "EVERYONE@", 9)) { + acep->ae_tag = ACL_EVERYONE; + acep->ae_id = ACL_UNDEFINED_ID; + gotid = 1; + } + if (gotid == 0) { + if (flag & NFSV4ACE_IDENTIFIERGROUP) { + acep->ae_tag = ACL_GROUP; + aceerr = nfsv4_strtogid(name, len, &gid, p); + if (aceerr == 0) + acep->ae_id = (uid_t)gid; + } else { + acep->ae_tag = ACL_USER; + aceerr = nfsv4_strtouid(name, len, &uid, p); + if (aceerr == 0) + acep->ae_id = uid; + } + } + if (len > NFSV4_SMALLSTR) + free(name, M_NFSSTRING); + + if (aceerr == 0) { + /* + * Handle the flags. + */ + flag &= ~NFSV4ACE_IDENTIFIERGROUP; + if (flag & NFSV4ACE_FILEINHERIT) { + flag &= ~NFSV4ACE_FILEINHERIT; + acep->ae_flags |= ACL_ENTRY_FILE_INHERIT; + } + if (flag & NFSV4ACE_DIRECTORYINHERIT) { + flag &= ~NFSV4ACE_DIRECTORYINHERIT; + acep->ae_flags |= ACL_ENTRY_DIRECTORY_INHERIT; + } + if (flag & NFSV4ACE_NOPROPAGATEINHERIT) { + flag &= ~NFSV4ACE_NOPROPAGATEINHERIT; + acep->ae_flags |= ACL_ENTRY_LIMIT_INHERIT; + } + if (flag & NFSV4ACE_INHERITONLY) { + flag &= ~NFSV4ACE_INHERITONLY; + acep->ae_flags |= ACL_ENTRY_ONLY_INHERIT; + } + if (flag & NFSV4ACE_SUCCESSFULACCESS) { + flag &= ~NFSV4ACE_SUCCESSFULACCESS; + acep->ae_flags |= ACL_ENTRY_SUCCESSFUL_ACCESS; + } + if (flag & NFSV4ACE_FAILEDACCESS) { + flag &= ~NFSV4ACE_FAILEDACCESS; + acep->ae_flags |= ACL_ENTRY_FAILED_ACCESS; + } + /* + * Set ae_extended. + */ + if (acetype == NFSV4ACE_ALLOWEDTYPE) + acep->ae_extended = ACL_EXTENDED_ALLOW; + else if (acetype == NFSV4ACE_DENIEDTYPE) + acep->ae_extended = ACL_EXTENDED_DENY; + else if (acetype == NFSV4ACE_AUDITTYPE) + acep->ae_extended = ACL_EXTENDED_AUDIT; + else if (acetype == NFSV4ACE_ALARMTYPE) + acep->ae_extended = ACL_EXTENDED_ALARM; + else + aceerr = NFSERR_ATTRNOTSUPP; + } + + /* + * Now, check for unsupported flag bits. + */ + if (aceerr == 0 && flag != 0) + aceerr = NFSERR_ATTRNOTSUPP; + + /* + * And turn the mask into perm bits. + */ + if (aceerr == 0) + aceerr = nfsrv_acemasktoperm(acetype, mask, owner, VREG, + &acep->ae_perm); + *aceerrp = aceerr; + if (acesizep) + *acesizep = NFSM_RNDUP(len) + (4 * NFSX_UNSIGNED); + return (0); +nfsmout: + return (error); +} + +/* + * Turn an NFSv4 ace mask into R/W/X flag bits. + */ +static int +nfsrv_acemasktoperm(u_int32_t acetype, u_int32_t mask, int owner, + enum vtype type, acl_perm_t *permp) +{ + acl_perm_t perm = 0x0; + + if (mask & NFSV4ACE_READDATA) { + mask &= ~NFSV4ACE_READDATA; + perm |= ACL_READ_DATA; + } + if (mask & NFSV4ACE_LISTDIRECTORY) { + mask &= ~NFSV4ACE_LISTDIRECTORY; + perm |= ACL_LIST_DIRECTORY; + } + if (mask & NFSV4ACE_WRITEDATA) { + mask &= ~NFSV4ACE_WRITEDATA; + perm |= ACL_WRITE_DATA; + } + if (mask & NFSV4ACE_ADDFILE) { + mask &= ~NFSV4ACE_ADDFILE; + perm |= ACL_ADD_FILE; + } + if (mask & NFSV4ACE_APPENDDATA) { + mask &= ~NFSV4ACE_APPENDDATA; + perm |= ACL_APPEND_DATA; + } + if (mask & NFSV4ACE_ADDSUBDIRECTORY) { + mask &= ~NFSV4ACE_ADDSUBDIRECTORY; + perm |= ACL_ADD_SUBDIRECTORY; + } + if (mask & NFSV4ACE_READNAMEDATTR) { + mask &= ~NFSV4ACE_READNAMEDATTR; + perm |= ACL_READ_NAMED_ATTRS; + } + if (mask & NFSV4ACE_WRITENAMEDATTR) { + mask &= ~NFSV4ACE_WRITENAMEDATTR; + perm |= ACL_WRITE_NAMED_ATTRS; + } + if (mask & NFSV4ACE_EXECUTE) { + mask &= ~NFSV4ACE_EXECUTE; + perm |= ACL_EXECUTE; + } + if (mask & NFSV4ACE_SEARCH) { + mask &= ~NFSV4ACE_SEARCH; + perm |= ACL_SEARCH; + } + if (mask & NFSV4ACE_DELETECHILD) { + mask &= ~NFSV4ACE_DELETECHILD; + perm |= ACL_DELETE_CHILD; + } + if (mask & NFSV4ACE_READATTRIBUTES) { + mask &= ~NFSV4ACE_READATTRIBUTES; + perm |= ACL_READ_ATTRIBUTES; + } + if (mask & NFSV4ACE_WRITEATTRIBUTES) { + mask &= ~NFSV4ACE_WRITEATTRIBUTES; + perm |= ACL_WRITE_ATTRIBUTES; + } + if (mask & NFSV4ACE_DELETE) { + mask &= ~NFSV4ACE_DELETE; + perm |= ACL_DELETE; + } + if (mask & NFSV4ACE_READACL) { + mask &= ~NFSV4ACE_READACL; + perm |= ACL_READ_ACL; + } + if (mask & NFSV4ACE_WRITEACL) { + mask &= ~NFSV4ACE_WRITEACL; + perm |= ACL_WRITE_ACL; + } + if (mask & NFSV4ACE_WRITEOWNER) { + mask &= ~NFSV4ACE_WRITEOWNER; + perm |= ACL_WRITE_OWNER; + } + if (mask & NFSV4ACE_SYNCHRONIZE) { + mask &= ~NFSV4ACE_SYNCHRONIZE; + perm |= ACL_SYNCHRONIZE; + } + if (mask != 0) + return (NFSERR_ATTRNOTSUPP); + *permp = perm; + return (0); +} +#else +/* + * Handle xdr for an ace. + */ +APPLESTATIC int +nfsrv_dissectace(struct nfsrv_descript *nd, struct acl_entry *acep, + int *aceerrp, int *acesizep, NFSPROC_T *p) +{ + u_int32_t *tl; + int len, gotid = 0, owner = 0, error = 0, aceerr = 0; + u_char *name, namestr[NFSV4_SMALLSTR + 1]; + u_int32_t flag, mask, acetype; + gid_t gid; + uid_t uid; + + *aceerrp = 0; + NFSM_DISSECT(tl, u_int32_t *, 4 * NFSX_UNSIGNED); + acetype = fxdr_unsigned(u_int32_t, *tl++); + flag = fxdr_unsigned(u_int32_t, *tl++); + mask = fxdr_unsigned(u_int32_t, *tl++); + len = fxdr_unsigned(int, *tl); + if (len < 0) { + return (NFSERR_BADXDR); + } else if (len == 0) { + /* Netapp filers return a 0 length who for nil users */ *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***